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ABSTRACT 
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learned and unresolved issues. Also included are five research 
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ego develcpment trajectories; an analysis of self-sustained learning 
and development; a study of career trajectories of women; research 
toward a taxonomy of alumnae generic and professional abilities; and 
an evaluation of professional abilities and student outcomes. 
Appendixes contain a description of the current student body at 
Alverno, a description of the longitudinal sample, charts of 
educational and developmental stages and relationships, a summary of 
statistics, a summary of generic abilities, an example of coding 
analysis of an essay, and 114 references . (JB) 
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UNDERSTANDING ABILITIES, LEARNING AND DEVELOPMENT THROUGH 



COLLEGE OUTCOMES STUDIES: 

1 

WHAT CAN WE EXPECT FROM HIGHER EDUCATION ASSESSMENT? 



Marcia Mentkowski, Glen Rogers, Deborah Deemer, Tamar Ben-Ur 

Judy Re i setter, William Rickards, Mary Talbott 

Office of Research and Evaluation 
ALVERNO COLLEGE 

This symposium examines the extent to which educational research that 
is internally driven, by institutionally defined purposes and 
questions, is valid and useful to the educational research community, 
as a way of addressing larger questions about teaching and learning. 

Recent state mandates for higher education assessment have generated a 
spate of studies within colleges and universities designed to collect 
information on college outcomes. 

What can we expect from these studies in higher education assessment? 
As educational researchers invested in our discipline or as 
institutional representatives doing assessment, we expect that 
assessment — carried out by and for an institution — will improve 
educational practice and student learning. 

But to what degree will we realize these expectations? The assessment 
movement currently is concerned with one aspect of this question, "To 
what degree can assessment benefit the institution that initiates it?" 
This question is of primary importance to those of us doing assessment 
because many of us are not satisfied what has been going on in the 
name of assessment. In Assessment for Excellence , Astin (1991) argues 
that "although a great deal of assessment activity goes on in 
America's colleges and universities, much of it is of very little 
benefit to either students, faculty, administrators, or institutions 
(p. ix)". Nevertheless, developing assessment practice designed for a 
particular institution's purposes to enhance educational benefits is 
clearly an important goal (Mentkowski, in press). 
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Symposium presented at the American Educational Research Association, 
Chicago, April 5, 1991. The authors acknowledge Joan Stark, University 
of Michigan, and Jon Wergin, Virginia Commonwealth University, for 
their role in this Division J (Post secondary Education) symposium, and 
for their review and critique of this paper. Stark, who is Director of 
the National Center for Research to Improve Postsecondary Teaching and 
Learning, set the context for the symposium and served as discussant. 
Wergin, who is Vice President of Division I (Education in the 
Professions) and a member of the Alverno Research and Evaluation 
Advisory Council, also served as discussant. Alverno 's Research and 
Evaluation Committee critiqued this paper: members are Zita Allen, 
Lucy Cromwell, Mary Diez, Austin Doherty, Georgine Loacker, Kathleen 
O'Brien, Timothy Riordan and Stephen Sharkey. Kathleen Schwan, Beverly 
Weeden, Lynn Chabot-Long and Rene Sisouphone contributed to the 
production of the paper. 
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Therefore, educational researchers doing assessment are asking (see 
Figure 1) : 

(1) To what degree do college outcomes studies contribute to an 
institution's purposes? 

Today's discussion will deal indirectly with this question, as it is 
incorporated in the first objective of this symposium. Today's 
discussion is designed to enable the AERA Division J membership to 
begin exploring this objective: 

(a) To discuss how AERA's Division J can best provide a forum 
and critique for such studies so as to influence the 
direction and quality of higher education assessment as an 
emerging field. 

The presenters at this table believe AERA can and does provide an 
excellent forum for this discussion. (In fact, Alverno faculty and 
research staff made 19 presentations at AERA from 1980 to 1990.) It is 
our practice to invite critique along the way, as we are designing and 
implementing various research and evaluation strategies, and after 
sets of results are in and interpretations made. Clearly, our 
research and evaluation team is a primary beneficiary from today's 
critique, as are other institutions and their representatives who have 
similar concerns (see Figure 1). 

But what should be the nature of the critique? What criteria should 
such studies aim to meet? By what evidence should such studies be 
judged? What are the standards to which such studies should be held? 
Because we, as AERA- J members, are concerned with these questions, we 
are invested parties in today's critique. 

Because the higher education assessment movement has created a new 
context for conducting educational research, there is a leadership 
role for AERA-J in ensuring the quality of higher education 
assessment. Many educational psychologists are involved in carrying 
out college outcomes studies in their own institutions. These studies 
represent a new source of hard money funding for educational research, 
badly in need of research dollars. 

However, such studies can potentially make other contributions in 
addition to examining an individual institution's quality, 
effectiveness, and validity. For educational psychologists, it seems 
unwise to overlook higher education assessment as a source for 
contributions to the philosophy, principles and practice of 
educational research. 

Consequently, we ask: 

(2) To what degree can college outcomes studies y: eld new knowledge 
re research/evaluation/measurement methods and practice, and 
illuminate its philosophical assumptions and values? 

A corresponding symposium objective is: 

(b) to critique the value of diverse methods for contributions 
to educational research philosophy/principles/practice; 

ERiC 
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Figure 1. Symposium questions, objectives and potential beneficiaries. 



Symposium Questions 



1. To what degree can college outcomes 
studies contribute to an institution's 
purposes? 



Objectives 



a. To discuss how AERA's Division J 
can best provide a forum and critique 
for such studies so as to influence the 
direction and quality of higher 
education assessment as an emerging 
field. 



Potential Beneficiaries 

• Institutions and their representatives 
engaged in institutional assessment. 

• AERA Division J membership. 

• Alverno research and evaluation 
team/committee. 



2. To what degree can college outcomes 
studies contribute new knowledge to 
educational research/evaluation/ 
measurement methods and practice, 
and illuminate its philosophical 
assumptions and values? 



b. To critique the value of diverse 
methods for contributions to 
educational research philosophy/ 
principles/practice. 



• Alverno research and evaluation 
team/committee. 

• Educational researchers developing the 
field of educational research. 

• Teachers, students and other 
practitioners of educational research. 



3. To what degree can college outcomes 
studies contribute new knowledge to 
discipline-based theories and methods 
in student/adult development, learning 
and abilities, which can form a partial 
base for undergraduate educational 
practice? 



c. To critique the value of diverse 
methods and results for building 
discipline-based theory/method. 



• Alverno research and evaluation 
team/committee; Aiverno faculty and 
staff. 

• Teachers and other advisors of college 
students. 

• Scholars in college student/ adult 
development, learning and abilities. 



4. How can college outcomes studies 
simultaneously c ontribute 
to a particular institution's purposes 
and to the more general purposes of 
educational research and post- 
secondary practice? 



d. To identify issues that studies 

conducted for purposes of institutional 
assessment will need to resolve and 
criteria studies will need to meet in 
order to contribute to postsecondary 
research and practice beyond a 
particular institution. 



• Alverno research and evaluation 
philosophy, principles and practice; 
Alverno frameworks: Educational 
philosophy, principles and practice. 

• Educational research community 
interested in improving the quality of 
its discipline and benefits of its 
profession for its higher education 
constituencies. 

• Institutions and their representatives 
interested in advancing higher 
education through institutional 
assessment and other forms of 
institutional scholarship and 
development 
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Potential beneficiaries of these contributions could include 
educational research colleagues who are engaged in similar purposes as 
the Alverno team, but more broadly, educational researchers who are 
developing the field itself, and teachers, students and other 
practitioners of educational research. 

It io also important to ask whether college outcomes studies can 
contribute to oar general understanding of student/adult development, 
learning and abilities. Can higher education assessment studies 
contribute to one of the key knowledge bases for educational practice? 

Consequently, we ask: 

(3) To what degree can college outcomes studies contribute new 
knowledge to discipline-based theories and methods in 
student/adult development, learning and abilities, which can 
form a partial base for undergraduate educational practice? 

Alverno researchers think college outcomes studies can make such a 
contribution, provided an institution has the development of student 
abilities, learning and personal growth as part of its mission. We 
are asking AERA- J to critique this assumption. 

Thus, a corresponding symposium objective is: 

(c) to critique the value of diverse methods and results for 
building discipline-based theory/method. 

Potential beneficiaries of these contributions could include those of 
us who are directly or indirectly involved in improving practice, 
because we are teachers or other advisors of college students. 
Benefits will be more likely for those of us who are educating a 
diverse population of traditional and nontraditional age women, a 
population Alverno serves. Large numbers of these women have recently 
entered higher education. Another audience for these contributions 
consists of scholars who are engaged in research in human development, 
learning, and abilities. 

Today's symposium will test the second question about contribution to 
educational research methods and the third question about contribution 
to discipline-based theory and methods through a discussion of diverse 
methods and results from a coherent approach to higher education 
institutional assessment that has been in place 15 years, and through 
an examination of five examples from complete and ongoing studies. 

In this emerging educational research mode, purposes, goals and 
methods for institutional assessment need to be coherent with those of 
a particular context. Does the primary emphasis on internal purposes 
and use of findings at a particular institution limit the benefit for 
external audiences? How might internal and external uses of 
information complement each other? This leads to a fourth question, 
which is our overall symposium theme: 

(4) How can college outcomes studies contribute to a particular 
institution's purposes and simultaneously contribute to the 
more general purposes of educational research and postsecondary 
practice? 
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Which criteria should euch studies meet to maximize broader benefits 
for educational research and postsecondary practice? How do we define 
"contribute to internal institutional purposes?" How do we 
define "contribute to more general purposes?" These questions are 
incorporated in our final, and most important symposium objective, 
which is: 

(d) to identify issues that studies conducted for purposes of 
institutional assessment will need to resolve and criteria 
studies will need to meet in order to contribute to 
postsecondary research and practice beyond a particular 
institution. 

Who will benefit from the critique implied in this objective? 
Institutions and their representatives engaged in institutional 
assessment, and the AERA Division J membership will benefit (see 
Figure 1) . 

The two discussants who will provide a response have qualifications 
that are particularly suited to such critique. They do provide 
different viewpoints. But more important, they can speak to two 
critical audiences for this symposium. The first audience is the 
educational research community interested in improving the quality of 
its discipline and benefits of its profession for its higher education 
constituencies. The chair and discussant, Joan Stark, responds out of 
her experience as Director of NCRIPTAL, a research center designed to 
meet a wide range of institutions' purposes. She is experienced in 
carrying out the purposes of a national center for research to improve 
postsecondary teaching and learning, which was charged with 
contributing research that could benefit educational practice in 
general. She can also take the perspective of ASHE, as she is a 
former president of that higher education organization. 

The other discussant addresses another critical audience for this 
symposium: institutions and their representatives interested in 
advancing higher education through institutional assessment and other 
forms of institutional scholarship and development. Jon Wergin was an 
NCRIPTAL evaluator, and has had to ask about contributions of research 
centers to teaching and learning practice. He is also directly 
involved in institutional assessment efforts. Thus, he is experienced 
in seeing the issues from the outside and the inside of an institution 
and can take an institutional perspective. He can also take the 
perspective of a current AERA Vice President. Jon Wergin is also a 
member of Alverno's Research and Evaluation Advisory Council, composed 
of individuals who from time to time, provide critique for the 
College's research and evaluation purposes and activities. 
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Alverno research and evaluation, philosophy, principles and practices 
will also benefit. (Alverno's Office of Research and Evaluation 
organized this AERA symposium as the fourth in a series of 
presentations dealing with this topic: AAHE Assessment Forum and the 
American Evaluation Association (Mentkowski, 1989), and the American 
Psychological Association (Mentkowski, 1990)). Inviting such critique 
and making these contributions is an explicit component of the 
College's mission (Alverno College, 1986). Our view of ourselves as 
professionals demands such critique and contribution. Thus, it is an 
explicit goal of the Office of Research and Evaluation to elicit 
constructive critique and to contribute to higher education research 
and evaluation. In turn, we expect to contribute to creating more 
generalizable models of adult abilities, learning and development (see 
Figure 2) . We work to contribute to emerging pictures of human 
potential, pictures of what it is possible for students to become. 
These pictures function as sources for goal setting, instructional 
strategies and assessment criteria for a student-centered institution. 

The Office of Research and Evaluation has been in place long enough 
(15 years) to enable us to review results from long-term and ongoing 
studies. The research team also has experience with a diversity of 
methods and theoretical frameworks. 2 

One might argue that reviews like today's are essential as a starting 
point into the topic of this symposium: What can we expect from 
higher education assessment? Other educational researchers involved 
in other higher education assessment programs such as the University 
of Tennessee-Knoxville, Northeast Missouri State University, Kean 
College, or Clayton State College could also offer a systematic review 
of their institution's research results in the context of today's 
questions. James Madison University, Miami University of Ohio, 
Mil] saps College and CEGEP (Colleges d'enseignement general et 
professionnel) in Montreal (Bateman, 1990) are candidates for such a 
review related to the topic of student/adult learning, development and 
abilities. 



2 While the presenters are working at the same institution and are part 
of a research and evaluation team, they represent the disciplines of 
educational psychology, developmental psychology, social psychology, 
sociology, evaluation and higher education administration. They were 
schooled in different theoretical frameworks and methods at seven 
different Universities (University of Wisconsin - Madison; University 
of Kentucky; University of Minnesota; University of Illinois; 
University of Arizona; Iowa State University; and the University of 
Wisconsin - Milwaukee) . The studies they will present today represent 
diverse theory and methods. This inter-disciplinary team works to 
ensure that any one study, even though it is grounded in a particular 
theory or method, has the benefit of the thinking of researchers from 
the other disciplines represented on the team. The research team 
consults with faculty in other disciplines in the institution through 
committees that meet for this and other purposes . ) 
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Figure 2 . Alvemo mission and Office of Research and Evaluation goals. 

Alverno Mission Office of Research and 

Evaluation Goals 

Alverno' s mission is the personal 
and professional development of 
women through education. This 
ijoal describes both our long-term 
and our daily pursuits, and we 
regard four activities as 
essential to both. 



Creating a Curriculum ; We 
organize learning so that it 
develops students' abilities, 
builds on a liberal arts 
foundation, is rooted in the 
Catholic tradition, accommodates 
the diverse needs of women, and 
is affordable for women of 
varied economic circumstances. 



Initiate and maintain research 
and evaluation as a concept 
and function at Alverno. 

Research and evaluate the 
quality, effectiveness and 
validity of the learning 
process. 



Creating a Community of Learning ; 
The common purposes that gathers 
Alverno faculty, staff, students 
and supporters is the pursuit of 
knowledge and development; of 
students' abilities. 



Contribute to creating more 
generalizable models of adult 
abilities, learning and 
development. 

Contribute to program, student 
and faculty development. 

Collaborate in ensuring the 
quality of various research 
and evaluation activities 
within the college. 



Creatine Ties to the Community ; 
Learning i squires relationships 
with business, industry and 
community institutions so that 
students prepare effectively to 
enter or continue in the world 
of work and fulfill the 
responsibilities of citizenship 
and service. 



Establish Alverno as an 
accountable educational 
institution in the local/ 
professional community. 
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Creating Relationships with 
Higher Education ; Faculty and 
staff elicit from colleagues 
constructive criticism of their 
teaching, scholarship and research 
on teaching and learning. In this 
way, they hold themselves responsible 
for a continuing contribution to the 
advancement of undergraduate education. 



Elicit constructive critique 
from colleagues, and 
establish Alverno as a 
contributor to higher 
education research and 
evaluation. 
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There are other strategies for asking these questions that can also 
provide a test of the benefits to postsecondary research and practice 
from institutional assessment. The literature review is a case in 
point. Indeed, Pascarella and Terenzini's (1991) recent analysis may 
provide a jumping off point for setting context, because they review 
college outcomes studies. We would, of course, have to separate out 
those studies conducted for purposes other than institutional 
assessment in our critique. As the assessment literature expands, 
this will become an important activity. Indeed, John Heywcod's 1989 
book on higher education assessment, Peter Ewell's 1984 book on 
self-regarding institutions, Ewell's 1985 book on assessing 
educational outcomes and his recent airticle (1991), are four such 
reviews (Alverno's institutional assessment efforts and those of other 
institutions are discussed in each of these sources . ) 

In sum, the presenters' examples, and the discussants' critique, can 
open the issues before us: What can educational research expect from 
higher education assessment? What can discipline-based theory and 
method in adult development, learning and abilities expect? How can 
AERA's Division J best influence the direction and quality of higher 
education assessment as an emerging field and what criteria should be 
met? Ultimately, what are the issues that need to be resolved in 
order for institutional assessment studies to contribute to 
postsecondary practice beyond a particular institution? This 
symposium examines the extent to which educational research that is 
internally driven, by institutionally defined purposes and questions, 
is valid and useful to the educational research community, as a way of 
addressing larger questions about teaching and learning. 

Let us now move to Figure la (an extended version of Figure 1) which 
is inserted in its fuller form once again at this point. This column 
"translates" each of the symposium questions into an "Alverno" 
question, since we are presenting today. Clearly, other institutions 
may ask their own questions in this column, and might ask them quite 
differently, given their own purposes and college outcomes vraUes. 
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UNDERSTANDING ABILITIES, LEARNING AND DEVELOPMENT THROUGH COLLEGE OUTCOMES STUDIES: WHAT CAN WE EXPECT FROM HIGHER EDUCATION ASSESSMENT? 1 



Symposium Questions 



Objectives 



1. To what degree can 
college oui >mcs 
studies coni bute to an 
institution's purposes? 



Potential Beneficiaries 



To discuss how AERA's 
Division J can best 
provide a forum and 
critique for such studies 
so as to influence the 
direction and quality 
of higher education 
assessment as an 
emerging field. 



Corresponding 
Alverno Questions 



Lessons Learned 



Unresolved Issues 



Which Criteria Apply? 



• Institutions and their 
representatives engaged 
in institutional 
assessment* 

• AERA Division J 
membership. 

• Alvcmo research and 
evaluation team/ 
committee. 



Do Alverno frameworks 
contribute to a general 
philosophy/principles/ 
practices of undergraduate 
education? (1 A) 

Do Alvemo research and 
evaluation efforts 
contribute to institutional 
purposes, and so support the 
contribution of Alvemo 
frameworks to general 
undergraduate education? (IB) 

How should Alvemo 

do institutional assessment? (1C) 



Major Theme 

How can college 
alumnae outcomes 
studies simultaneously 
contribute to a particular 
institution's purposes 
and to the more general 
purposes of educational 
research and 
postsccondary practice? 



To identify issues that 
studies conducted for 
purposes of institutional 
assessment will need to 
resolve and criteria 
studies will need to 
meet in order to 
contribute to 
postsecordary 
research and practice 
beyond particular 
institution. 



Alvemo research and 
evaluation philosophy, 
principles and practice; 
Alvemo frameworks: 
Educational philosophy, 
principles and practice. 

Educational research 
community interested in 
improving the quality of 
its discipline and 
benefits of its profession 
for its higher education 
constituencies. 

Institutions and their 
representatives 
interested in advancing 
higher education through 
institutional assessment 
and other forms of 
institutional scholarship 
and development. 



How can A ,verno college 
outcomes studies 
simultaneously contribute to an 
institution's purposes and 
to the more general purposes 
of educational research and 
pos Secondary practice? 



• Contextually valid findings 
contribute to general theory 
and practice when they link 
practice-based frameworks 
with discipline-based theory 
in a dynamic setting. 

Interdisciplinary research 
teams apply criteria from 
various theories and methods, 
for various purposes. This 
creates paradoxes. 



• What are principles of action 
and applied research that is 
problem-driven and practice 
based? 

• What are principles of 
institutional assessment that is 
also interactive, 
interdisciplinary, integrative, 
and intends simultaneous 
internal and external 
contributions? 

• Which criteria apply? How 
will these shape institutional 
assessment? 



Cohe/ence/diversity among 
purposes, designs, methods 

Utilization/gencralizability of 
methods, results 

Validity/reliability of 
measurement 

Rigor/feasibility of method 



M. Mentkowski, 0. Rogers, D. Deemer, T. Ben-Ur, J. Reisctter, 

W. Rickards, M. Talbott. Symposium presented at the 

A -^- ! -an Educational Research Association, Chicago, April 5, 1991. 

aye 



. 4 Page 1 
14 



Symposium Questions 



Objectives 



Potential Beneficiaries 



Corresponding 
Alverno Questions 



Five Alverno Examples 



Lessons Learned 



Unresolved Issues 



2. To what degree can 
college outcomes studies 
contribute new knowledge to 
educational research/ 
evaluation/measurement 
methods and practice, and 
illuminate its philosophical 
assumptions and values? 



b. To critique the value of 
diverse methods for 
contributions to educational 
research philosophy/ 
principles/practice, 



Alverno research and 
evaluation team/ 
committee. 

Educational researchers 
developing the field of 
educational research. 

Teachers, students and 
other practitioners of 
educational research. 



Do Alverno educational 
research and evaluation 
firameworks(philosophy/ 
principles/practice) contribute 
to general principles for 
educational research and 
evaluation practice? 



Longitudinal analysis 
ofchangeasaresultof 
curriculum (qualitative, 
quantitative) (I, II, ID) 

Analysis of professional 
/alumnae abilities (IV/V) 

Evaluation of general 
education and the major 
field (V) 



Multiple frameworks and 
methods yield a useful, 
complex picture of college 
outcomes. 

An institution's educational 
frameworks shape a 
conceptual base for 
assessment 

Contributions to methods can 
occur when eductions! 
purposes and research 
methods are inseparable. 



• How achieve long- and 
short-term benefits 
simultaneously? 



• How will different conceptual 
bases for assessment impact 
college outcomes studies? 



What institutional processes 
link purpose and method? 
How does method change as 
a result? 



3. To what degree can college 
outcomes studies contribute 
new knowledge to 
discipline-based theories 
and methods in student/ 
adult development, learning 
and abilities, which can 
forma partial base for 
undergraduate educational 
practice? 



c. To critique the value of 
diverse methods and results 
for building discipline-based 
theory/method. 



Alverno research and 
evaluation team/ 
committee; Alverno 
faculty and staff. 

Teachers and other 
advisors of college 
students. 

Scholars in college 
student/adult 
development, learning 
and abilities. 



What broad patterns 
describe development during 
college and afterwards, and 
how are these patterns 
related to curriculum? 



What learning outcomes 
describe development, and 
what curricular elements 
cause it from the student's 
perspective? 



Cognitive, moral and ego 
development trajectories (I) 



Self-sustained learning 
and development (II) 



Developmental trajectories 
differ across domains and 
timeframes. 



Knowledge and performance 
are linked in student 
constructions and attributed to 
curricular elements; 
understanding criteria leads to 
self -sustained learning. 



Who changes and why? 
What are best methods for 
analyzing intra- and 
inter-individual change 
patterns? 



» What student perspectives and 
curricular elements relate to 
gains in cognitive, moral and 
ego development? 



1 : 1 



Do women graduates realize 
their goals? 



How do graduates perform in 
personal and professional 
domains? 



How do abilities derived 
from studies of outstanding 
professionals link to 
evaluation of student 
outcomes in the major field? 



Career trajectories of 
women (HI) 



Alumnae generic 
abilities (IV) 



Professional abilities and 
student outcomes (V) 



Older and younger women 
achieve management and 
professional positions after 
college. 



Some generic abilities 
distinguish effective alumnae 
performance. 

Curriculum development is 
informed by a dynamic 
relationship among coherent, 
diverse research and 
evaluation strategies. 



» What alumnae measures 
consider the intersection of 
personal and professional, 
public and private 
contributions? 

• How well do generic ability 
codes cross a wide range of 
personal and professional 
activities? 

How does personal 
development interact with 
disciplinary outcomes? What 
form does personal and 
professional integration take? 
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SYMPOSIUM QUESTION ONE: TO WHAT DEGREE CAN COLLEGE OUTCOMES STUDIES 
CONTRIBUTE TO AN INSTITUTION'S PURPOSES? 

Alverno Question 1 A, B and C: 

Question 1 A: Do Alverno Frameworks (Educational 

Philosophy/Principles/Practice) Contribute to a General 
Philosophy/Principles/Practice for Undergraduate Education? 

Question IB: Do Alverno Research and Evaluation Efforts 
Contribute to Institutional Purposes, and so Support the 
Contribution of vlverno Frameworks to General Undergraduate 
Education? 

Question 1 C: How Should Alverno Do Institutional Assessment? 

The first section described the symposium questions and objectives, 
and why the participants organized the symposium. In the next 
paragraphs, we first show that it is part of Alverno 's mission and the 
Office of Research and Evaluation goals to examine whether Alverno 
frameworks contribute, and college outcomes studies support this 
larger contribution. Then we outline a series of questions that are 
often posed to Alverno by external audiences, as they question the 
potential of Alverno frameworks to contribute outside the institution. 
Finally, we cite several references for those persons interested in 
how Alverno does institutional assessment. He also refer the reader 
to our thinking on how we should do it (Mentkowski, 1989; 1990) . We 
close with a recap on the symposium questions. 

Now we turn to the potential contributions inherent in the symposium 
questions and objectives as these are diagrammed in Figure 3. Figure 
3 is a sort of map of the relationships we are discussing in the 
symposium. 

o Alverno contributions to general undergraduate education 

(Question 1 A) and Alverno 's research and evaluation efforts, as 
they contribute to institutional purposes, and so support this 
larger contribution (Question 1 B); 

o Contributions of Alverno 's research and evaluation frameworks to 
general research and evaluation philosophy, principles and 
practice (Question 2); 

o Contributions to Alverno 's research and evaluation methods and 
results to discipline-based theory and results (Question 3); 
and 

o Contributions that simultaneously meet internal and external 
needs and expectations (Question 4) . 
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-iods/results to research/evaluation/aeasuresent practice, and 
discipline-based theory /sethod in student/adult development, learning 
and abilities. 



ALVERNO FRAMEHORKS: 

EDUCATIONAL PBILOSOPHY/PMNCIPLBS/PRACTICE 

• Liberal arts/professional 

• Student-centered 

• Outcome-oriented 

• Coherent, developmental curricului 

• Ability-based, via the disciplines 

• Experiential learning 

• Assessment-as- learning for 
individual student developient { 
credentialing, prograi evaluation 



Question 1A 



Question 



Institutional Developient/ 
Effectiveness 




Question 3 



Question 



ALVERNO FRAMEHORKS: 

RESEARCH AND EVALUATION PHILOSOPHY/PRINCIPLES/PRACTICE 

• Educational framework-driven institutional assessient 

• Multi-level triangulated designs with mltiple 
internal and external comparisons 

• Collaborative, interdisciplinary office, teai, 
committee 

• Interactive collaboration with faculty /staff 

• Sustained student/aluma participation and benefits 

• Demonstrating quality /validity /effectiveness of 
learning process 

• Describinq/ascribing development, abilities, learning 

• Longitudinal analysis of change as a result of 
curricului (qualitative, quantitative) 

• Analysis of professional/alumnae abilities 

• Evaluation of general education and the major field 

• Teacher-as-researcber/inquirer studies 

• Evaluating/validating assessment-as-learning 

• Contextual validity definition 

• Strategies for validating faculty-designed 
performance assessment measures 

• Defining criteria for "good" assessment 



Question 2 

< 



General Philosophy/ 
Principles/Practice 
for Undergraduate 
Education 

Learning by Doing 
Core Curriculum 
Individualized Instruction 
Education for Development 
Collaborative Learning 
Student Outcomes Assessment 
Performance Assessment 
Interdisciplinary Studies 
Nulticultural Curriculum 






Pictures of 
Human Potential: 

Discipline-based 
theory and method 
in student/adult 

• Development 

• Learning 

• Abilities 



General Philosophy/ 
Principles/Practice for 
Higher Education 

Research and Evaluation 
• Educational research 
• College outcomes studies 
• Curriculum outcomes 
studies 
Educational evaluation 
Educational measurement 
Institutional research 
Institutional assessment 
Institutional scholarship 
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Figure 3 shows these potential contributions. Question 1 A asks 
whether Alverno educational frameworks contribute to general 
undergraduate educational philosophy, principles and practices, and 
whether Alverno' s institutional assessment program meets Alverno 's 
purposes (Question IB), and so undergirds this contribution. Does 
Alverno research and evaluation that tests, investigates, and examines 
Alverno 's philosophy, principles and practice contribute to the 
credibility, effectiveness and validity of Alverno 's frameworks in the 
arena of undergraduate education? We say yes, because we conduct 
these activities, in part, through comparisons with external 
theoretical frameworks. 

We stated earlier that it is part of Alverno 's mission and the Office 
of Research and Evaluation goals is to show that such contributions 
are occurring. We now turn to a more explicit discussion of this 
point for the reader who may be interested in Alverno 's mission and 
the kinds of evidence that can be cited to show that Alverno has made 
some progress toward the elements of that mission relevant to today's 
symposium. 

Alverno 's Mission 

One component of Alverno *s mission is to elicit constructive critique, 
and to contribute to the advancement of undergraduate education (see 
Figure 2). 

Creating relationships with Higher Education: Faculty and 
staff elicit from colleagues constructive criticism of their 
teaching, scholarship and research on teaching and learning. 
In this way, they hold themselves responsible for a 
continuing contribution to the advancement of undergraduate 
education (Alverno College, 1986, Chapter 1, pp. 4-5). 

Consequently, one goal of the Office of Research and Evaluation is to 
elicit constructive criticism from colleagues and to establish Alverno 
as a contributor to higher education research and evaluation in order 
to support this broader institutional goal. 

The number of citations in the literature, collaborations, and 
consultations with other institutions suggest some progress toward 
this broad institutional goal. For example, since 1973, there have 
been a total of 2,796 individuals from 894 institutions who have 
visited Alverno for at least a day or up to 10 days for in-house 
workshops. Since 1978, 20,132 copies of books about Alverno' s 
philosophy and educational frameworks have been disseminated, 
excluding reprints or Office of Research and Evaluation publications. 
In 1990 alone, 4,278 publications (including reprints but excluding 
Office of Research and Evaluation publications) were disseminated. 

The Office of Research and Evaluation report similar documentation on 
the degree to which the Office met similar goals from 1977 to 1987 
(see second edition of Mentkowski & Doherty, 1983 revised 1984). The 
Office disseminated 19,800 copies of five major articles and chapters 
developed from the research outcomes that were also distributed 
externally by outside publishers. 
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The Office created 68 publications and made 170 presentations. It 
responded to 797 requests by mail or during telephone consultations, 
and subsequently mailed 2,140 publications in response. Research 
outcomes were described or cited in 14 news articles and at least 56 
outside publications. From 1977 to 1987, we reached over 1,895 
institutions and representative departments in all 50 states and 29 
countries through presentations, together with countless publications 
distributed during presentations or mailed upon request. 

This documentation — and in particular, the publications and refereed 
presentations such as this symposium — is some evidence of eliciting 
critique and of contribution related to that part of the college's 
(and the Office of Research and Evaluation's) mission to examine 
whether and how Alverno frameworks, namely, educational 
philosophy/principles/practice contribute to general 
philosophy /principles/practice for undergraduate education (see 1 A in 
Figure 3) . It is also some evidence that the research and evaluation 
efforts support this larger contribution (see 1 B in Figure 3). What 
do we mean by Alverno frameworks? 



Alverno Educational Frameworks: Some Examples 

Various elements of Alverno 's educational frameworks: its philosophy, 
principles and practice, are described and discussed in Alverno 
literature. Today's purpose is not to illuminate these elements, but 
rather to enumerate them, so they can serve as a backdrop for a 
discussion of methods and results from the studies discussed today. 

These elements, listed in Figure 3, include: 

o liberal arts/professional 

o student-centered 

o outcome-oriented 

o coherent, developmental curriculum (Read & Sharkey, 1985) 

o ability-based, via the disciplines (Alverno College Faculty 
1976 revised 1985; Earley, Mentkowski & Shafer, 1980; 
Loacker & Palola, 1981; Loacker, Cromwell, Fey & Rutherford, 
1984; Read, 1980) 

o experiential learning (Doherty, Mentkowski & Conrad, 1978; 
Hutchings & Wutzdorff , 1988) 

o assessment-as-learning for individual student development, 
credent ia ling, program evaluation, and so on (Alverno 
College Faculty, 1979 revised 1985; Loacker, 1988; Loacker, 
Cromwell & O'Brien, 1986; Mentkowski & Loacker, 1985). 
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General Educational Frameworks: Some Examples 

Alverno is not alone in developing distinct educational frameworks. 
Several distinctive educational philosophies/principles/practices have 
emerged in the literature or are observed in practice at various 
institutions. 

The higher education literature cites certain undergraduate 
institutions that have been recognized from time to time over the 
decades as places where a recognizable educational philosophy, 
principles and practice have been realized (e.g. Harvard University, 
University of Chicago, Hampshire College, Reed College, University of 
California-Irvine; University of California-Santa Cruz; Evergreen 
College, and so on) . 

Further, distinctive general educational practices have emerged in 
higher education that have been developed and adapted by various 
individuals or institutions institutions. 

A partial listing in Figure 3 is as follows: 

o learning by doing 

o core curriculum 

o individualized instruction 

o education for development 

o collaborative learning 

o student outcomes assessment 

o performance assessment 

o interdisciplinary studies 

o multicultural curriculum 

Question 1 A: Do Alverno Educational Frameworks Contribute? 
Question IB: Do Alverno Research and Evaluation Frameworks Contribute? 
How Do External Audiences as These Questions? 

In our experience, before external audiences ask whether Alverno 
frameworks contribute to general undergraduate philosophy, principles 
or practice, they ask three questions of the institution. The first 
is a question of credibility, which often implies a judgment of 
demonstrated quality. A second question concerns effectiveness. The 
third is one of validity. Alverno 's research and evaluation efforts 
are one way the College responds to these questions. 
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In the following series of questions, "it" refers to any aspect of 
Alverno philosophy, principles or practice of interest to an external 
audience. 

o Is it credible? Can it be done? Are you actually doing it? To 
what degree are you as an educational institution actually doing 
what you put forth as your philosophy, principles and practice? 

(Alverno faculty and staff host semi-annual Visitation Days 
and annual workshops; organize and facilitate 
multi-institution consortia (three externally funded ones 
since 1983); and engage in presenting and publishing.) 

o Is it effective? Does this curriculum work well? 

(Effectiveness at Alverno rests on sets of institutionalized 
evaluation processes that ensure internal monitoring, 
revision and evaluation. The assessment-as-learning process 
that generates continuous data on student performance, and 
departmental review processes are just two examples. ) 

o Is it valid? Does this learning process cause student outcomes? 
What is the relationship of the institution's educational 
frameworks to outcomes? 

(Alverno chooses student/alumnae outcomes as the criterion, 
because student learning is at the heart of and central to 
the mission of the institution and the primary criterion for 
its effectiveness. "Do Alverno student outcomes meet 
Alverno 's internal criteria and those external criteria that 
the college judges relevant to its purposes and mission? Do 
students and alumnae achieve their potential as human 
persons?") 

Then comes the question of contribution to general undergraduate 
education (See Question 1 a in Figure 3): 

o To what degree has this institution' s frameworks contributed to 
higher education's educational philosophy, principles and 
practices? 

(Alverno documents its consulting activities and its 
citations in the external literature. ) 

Often, the way this question is asked takes another form: "What other 
institutions have inplemented the Alverno frameworks?" We prefer the 
former rendition of this question, because it does not assume that the 
test of contribution to a general framework is that another 
institution "copy" or "imitate" Alverno philosophy, principles or 
practices. For us, requirements of contextual validity preclude 
copying or imitating. Rather, contribution to general practice rests 
on inferences about the degree to which elements of the educational 
philosophy, principles or practices have been helpful or stimulating 
to another higher education institution which is developing a 
curriculum. This other curriculum is expected to be designed in view 
of the characteristics of the setting and population of that other 
institution. 
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Is responding to these questions from external audiences part of 
Alverno's mission? Yes. At Alverno, questions of quality, 
effectiveness, and validity are of primary concern from an 
institutional development perspective. From the beginning, Alverno 
has believed that the quality, effectiveness, and validity of 
frameworks in the institution, irrespective of questions raised by 
outside audiences, is critical to the further development of the 
frameworks themselves. 

One might argue that the contributions from the Alverno frameworks can 
take place without any kind of extensive research and evaluation of 
these frameworks beyond what occurs as a result of assessing 
individual student learning (Alverno College Faculty, 1979 revised 
1985; Loacker, Cromwell & O'Brien, 1986) and using the results to 
improve curriculum in an ongoing way. Most of us can recall examples 
of educational ideas — sometimes these are called fads — which sweep 
the educational community, even though they may have little 
justification as developed through either research or practice. 

One way such credibility, effectiveness, and validity can be examined 
is through conducting extensive examinations of these frameworks 
through studies of student and alumnae outcomes of the learning 
process, and inviting ongoing critique from one's various 
constituencies. 

More specifically, Alverno's research and evaluation office that 
tests, investigates, and examines Alverno's philosophy, principles and 
practice, has this goal: Contribute to the quality, effectiveness and 
validity of / Iverno's learning process (Figure 2). 

Once questions of credibility (or quality) , effectiveness and validity 
are dealt with, external audiences ask questions that suggest they 
themselves are beginning to or are already working at similar 
curriculum development efforts. Or they may be working at developing 
institutional assessment, for example. Then these questions surface: 

o How did you do it? How can it be done? o What lessons have you 
learned? 

o What about it might work anywhere else? Where might it work? 
How could it be adapted and still maintain its essential 
qualities? 

o If it's like this (e.g., coherent, developmental, complex), can 
I use it? What aspects can I use? 

o Has anyone elae taken it up? 
o philosophy 
o principles 
o practice 
o examples 

o Is it different? o Is it better? 
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Is dealing with these questions part of Alverno's mission? Yes. 
Alvemo expects to contribute to the advancement of undergraduate 
education and learn from colleagues engaged in similar efforts. 
Alverno's Office of Research and Evaluation is expected to establish 
Alvemo as a contributor to higher education research and evaluation, 
and so support these broader efforts. 

The external audience questions listed above are also asked of 
Alverno's research and evaluation philosophy, principles and 
practices. In addition, external audiences ask: 

o Can research and evaluation studies conducted at one college 
"add up to anything" across colleges? Can such studies speak to 
or address issues being raised in different contexts? 

To recap, Alverno's primary mission is the personal and professional 
development of women through education. Thus, faculty create a 
curriculum and community of learning engaged in the pursuit of 
knowledge and development of students' abilities. Therefore, an 
Office of Research and Evaluation goal is to contribute to more 
generalizable models of adult abilities, learning and development. It 
is the question of external contribution that is of issue here, rather 
than questions listed above that deal with how to do and develop 
institutional assessment. However, the next section is included for 
the reader who is asking about Question 1 C: How does — and should 
— Alvemo do institutional assessment? 

Question 1 C: How Should Alvemo Do Institutional Assessment? 

For many of us in higher education assessment, the major challenge is 
to demonstrate that institutional assessment can contribute to the 
improvement of programs in a particular institution. We are 
interested in how we should do institutional assessment. Symposia on 
this topic are regular program entries in AAHE's Assessment Forum held 
annually since 1985. 

Therefore, a list of citations follow for the reader interested in how 
Alvemo does institutional assessment, what the overall results are, 
and how findings are utilized in general. The following references 
provide such a description, and are available in the order form 
attached to this paper. Several references synthesize our research 
and evaluation philosophy, principles and practice (Mentkowski, in 
press; Mentkowski, 1988a; Mentkowski & Doherty, 1983 revised 1984; 
Mentkowski & Doherty, 1984; Mentkowski & Loacker, 1985; Mentkowski and 
Rogers; Read, 1985; Rogers, 1988; Talbott, 1989). 

The above mentioned citations describe the following: 

o findings which illustrate how we do research and evaluation and 
how we meet goals in relation to Alverno's mission and thus 
support Alverno's contributions to general undergraduate 
education. 
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o the organization and funding of the Office, the nature of Office 
goals and activities, descriptions of the complete set of 
research and evaluation frameworks, including designs, methods , 
results, and their utilization; the organization of the 
Research and Evaluation Committee: how this gran? of senior 
faculty work to refine research and avaluation frameworks; its 
role and function; and Committee contributions. 

How should Ai/erno conduct institutional assessment? We have 
discusse* the issues involved in establishing the validity and 
intecr- .f higher education assessment (Mentkowski, 1989), and the 
import of establishing a conceptual base for assessment 
(Mentkowski, 1990a) . In these discussions, we have explored the range 
of criteria that might be applied to higher education assessment 
studies, and the difficulties in meeting these criteria. For the 
purposes of today' s symposium, we are asking whether our institutional 
assessment studies can meet the criteria inherent in meeting 
institutional purposes (which include creating information that can be 
used to improve programs, or utilization criteria, for example) and 
those inherent in meeting purposes of external audiences (which 
include creating information that can be used to advance undergraduate 
education, or generalizability criteria, for example) . 

Thus, one internal test of the "goodness H of the studies is the extent 
to which the findings are actually used by the faculty to inform the 
educational frameworks of the College. Do results challenge, inform, 
assist in refinement, or promote the improvement of Alverno 
philosophy, principles and practices? In short, are the findings used 
by the faculty to improve the curriculum? Sources cited earlier 
describe our efforts to meet the criterion of internal contributions, 
as our Office and our Research and Evaluation Committee work to meet 
Alverno purposes. 

In the next section, Symposium Question Two, we briefly enumerate and 
describe Alverno frameworks for educational research and evaluation, 
and cite examples of potential contributions to research and 
evaluation methods and practice. 
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SYMPOSIUM QUESTION TWO: TO WHAT DEGREE CAN COLLEGE OUTCOMES STUDIES 
YIELD NEW KNOWLEDGE RE RESEARCH/EVALUATION/MEASUREMENT METHODS AND 
PRACTICE, AND ILLUMINATE ITS PHILOSOPHICAL ASSUMPTIONS AND VALUES? 

Alverao Question Two: Do Alverno Educational Research and Evaluation 
Frameworks ^ Philosophy /Principles/Practice) Contribute To General 
Principles for Educational Research and Evaluation Practice? 

Synopsis 

Marcia Mentkowski, Professor of Psychology and Director of the Office 
of Research and Evaluation at Alverno provides an overview of the 
coherence and multiplicity inherent in the approaches and strategies 
that characterize research and evaluation at the College. She 
identifies some examples of potential contributions to general 
research and evaluation practice from these integrated, yet diverse 
approaches and strategies. Then she sets the stage for the five 
examples of potential contributions to student/ar , *'lt development, 
learning and abilities presented by Office of Research and Evaluation 
staff in the next section. She outxines those examples, and briefly 
characterizes the setting, population and samples for the five 
examples. 

How is Contribution Defined and What is Evidence? 
Examples of Potential Methods Contributions 



Alverno frameworks: educational research and evaluation 

Alverno publications cited in the last section discuss Alverno 
research and evaluation philosophy, principles and practice. Figure 3 
(see lower, left-hand box) lists some of primary approaches and the 
accompanying strategies Alverno uses to conduct research and 
evaluation studies. 

It has been our experience that many of these approaches had to be 
developed as we went along. While we were able to draw on some 
existing methods, instruments, and practices, we consider the 
following approaches to be examples of methods we are developing, and 
to which our colleagues in institutional assessment are contributing 
as they engage in similar work. Because these have been discussed 
elsewhere by us and in the external literature, we will only briefly 
deal with these today. 
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These overall approaches are listed below, in order to set some 
context for five examples of contributions that will soon follow, and 
to identify three of the strategies that are used in the five 
examples. 

o Educational framework-driven institutional assessment 

o Multi- level triangulated designs with multiple internal and 
external comparisons 

o Collaborative, interdisciplinary office, team, committee 

o Interactive collaboration with faculty/staff 

o Sustained student/alumna participation and benefits 

o Demonstrating quality/validity/effectiveness of the learning 
process 

o Jescribing/ascribing development, abilities, learning 

o Longitudinal analysis of change as a result of curriculum 
(qualitative, quantitative) 

o Analysis of professional/alumnae abilities 

o Evaluation of general education and the major field 

o Teacher-as-researcher/inquirer studies 

o Evaluating/validating assessment-as-learning 

o Contextual validity definition 

o Strategies for validating faculty-designed performance 
assessment measures 

o Defining criteria for "good" assessment 

for example: 

o Coherence/diversity among purposes, designs, methods 
o Utilization/generalizability of methods/results 
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Our general approaches 

A conceptual base is important for assessment (Mentkowski, 1990a) . 
One of our general approaches can be described as educational 
framework-driven (theory-driven, goal-driven, problem-driven) 
"institutional assessment" (see Figure 3). 3 



Our institutional assessment philosophy argues that methods should be 
coherent with Alverno educational frameworks and theories. Thus, 
educational purposes and research and evaluation questions and methods 
are inseparable. Institutional assessment processes are grounded in 
Alverno educational philosophy, "bootstrapped" together with Alverno 
principles and refined in the context of changing educational 
practices, as well as changes in the setting (i.e. changes in major 
field, or characteristics of the student population) . This 
"interactive" research mode, this interdependent character of 
Alverno 's institutional assessment, creates a research and practice 
relationship that consists of a fabric of relationships that are woven 
together, where one can no longer speak of a linear, causal dimension 
of "research influencing practice" or "practice influencing research. " 
Educational purpose and inquiry methods are intertwined and coherent. 

As stated earlier, there are various elements of Alverno frameworks 
that are reflected in our educational research and evaluation 
philosophy, principles and practices, that will frame our approaches 
to institutional assessment. Because of these frameworks, these 
approaches and strategies simultaneously reflect concerns for 
coherence and diversity. 

For example, Alverno is a liberal arts college with emphasis on the 
personal and professional development of its students, and its 
curriculum builds on a liberal arts foundation. Consequently, 
assessment approaches are expected to incorporate multiple 
perspectives, multidisciplinary approaches, and multiple comparisons, 
with special attention to choosing external frameworks that open 
Alverno frameworks to other ones. This diversity and multiplicity is 
expected to characterize institutional assessment approaches and 
strategies. 



3 We are now using the term "institutional assessment," and working to 
define it more broadly, because it is current in higher education, and 
because our work has been widely cited as an example of institutional 
assessment. Within our institution, however, we reserve the term 
"assessment" for another approach that involves all of our students, 
that is, our faculty-designed assessment-as-learning processes for 
individual development (Alverno College Faculty, 1979 revised 1985; 
Loaoker, Cromwell & O'Brien, 1986; Mentkowski & Loacker, 1985)). 



UNDERSTANDING ABILITIES, LEARNING AND DEVELOPMENT 1991 AERA SYMPOSIUM 



Because the mission is the personal and professional development of 
its students , research and evaluation questions reflect a 
student-centered institution and a concern with whether and how each 
individual student demonstrates this development. Thus, student 
outcomes (student/alumna development, learning and abilities) are the 
"content" of our research and evaluation studies. Alverno educational 
frameworks include a coherent, developmental, ability-based curriculum 
with special attention to experiential, self -sustained learning and 
assessment-as-learning. Student outcomes of the curriculum are the 
focus of the institutional assessment enterprise. 

This student-centered purpose of the institution means that 
information from institutional inquiry has a central purpose: to 
enhance student development, learning and abilities. Information must 
be both useful and general. At the program or institutional level, 
information indirectly benefits individual students. But clearly, 
information is expected to be used for student benefits. At the same 
time, the broader picture of student achievement that emerges is 
multifaceted and collective, a backdrop against which faculty can 
interpret an individual student's growth. Pictures that accrue from 
aggregated sets of information over time are expected to inform 
curriculum development, but also to question the philosophy and 
principles upon which it is based. Still, these collective pictures 
should be easily transformed into intra- and inter-individual patterns 
that do not lose sight of the individual student's development. The 
pictures are impressionistic in that, as one steps away, a holistic 
scene appears. As one looks more closely, each dab of paint, each 
individual color, each brush stroke is evident. 

This developing conceptualization of institutional assessment has 
emerged at Alverno over the years, and its primary purpose is 
research, evaluation and validation of student outcomes. It has 
necessitated the development of multi-level, triangulated designs for 
research and evaluation of student/alumna outcomes that call for 
multiple, internal and external comparisons (Mentkowski & Doherty, 
1983 revised 1984; Mentkowski & Loacker, 1985). (See Figure 4, 
"Multi-level triangulated validation design with multiple internal and 
external comparisons" . ) We have selected, adapted or developed 
multiple instruments and methods drawn from our own and external 
theoretical frameworks. 
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Figure 4 . Multi-level triangulated validation design with multiple internal and external comparisons. 
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Our approach has also called for the development of a collaborative, 
inter-disciplinary research office, an interdisciplinary research and 
evaluation team that does not teach classes or advise students. We 
have an interdisciplinary research and evaluation committee made up of 
senior faculty and administrators and chaired by the Director of the 
Office. 

To carry out educational framework-driven institutional assessment, we 
have developed interactive, interdisciplinary processes that are 
effective in developing a collaborative interplay that engages faculty 
questions and contributions (Mentkowski, 1988b) . By engaging the 
whole faculty in question-asking (Mentkowski, in press), and by 
tapping existing faculty groups related to particular issues, the 
Office formulates research questions. Nor does faculty leadership and 
investment stop there. Over the years, faculty have served in various 
capacities as adjunct members of the research team, as advisors, as 
interpreters of results, and so on. The Research and Evaluation 
Committee, comprised of senior-level faculty and administrators, is a 
springboard and interpreter at the institutional level of 
question-asking and interpretation. Findings and their 
interpretations are an outcome of this interplay all the way through 
the process, from research question, through data collection and 
analysis, interpretation of results, and making meaning out of the 
results for curriculum development. 

We have been active in creating strategies for question-asking that 
work to integrate research, evaluation and practice at the national 
level, in that we actively co-lead and support the AAHE Research Forum 
(Mentkowski & Chickering, 1987), which has generated a research agenda 
each year since 1986. This involvement ensures that Alverno's 
research and evaluation activities are in tune with national questions 
and issues that educators feel should be the subject of inquiry. 

We have also created methods that result in sustained participation of 
samples of students and alumnae in research and evaluation activities. 
Key elements are providing rationales and immediate benefits (such as 
feedback on instrument results) to these participants (Mentkowski, 
1989; Mentkowski & Strait, 1983; Reisetter & Sandoval, 1987). 

Research and evaluation goals 

Recall that two of our research and evaluation goals are to 
demonstrate quality, validity, and effectiveness of the learning 
process, and to describe and ascribe student and alumna development, 
abilities and learning (Figure 2). In order to do this, we have 
employed a number of more specific strategies. Three are the focus of 
today's examples. 

Strategies that flow from research and evaluation goals 

A first strategy is the longitudinal analysis of change as the 
result of curriculum (Mentkowski, 1990b; Mentkowski & Strait, 1983), 
which is a strategy for both research and evaluation of the broad 
outcomes of college. This strategy provides for more short-term 
evaluation benefits for evaluation of the curriculum at its earlier 
phases when one is generating information on current students. It 
provides for more long-term research benefits at its later phases as 
it works to describe longitudinal antecedents of alumnae abilities, 
learning and development. Our longitudinal strategies employ both 
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quantitative and qualitative methods. Further, these strategies have 
used instruments and methods that are drawn from a variety of 
theoretical frameworks in cognitive development, learning styles, and 
broad abilities or competences. Because we draw on a range of 
theoretical frameworks that relate to faculty educational frameworks 
in student development, learning and abilities for external 
comparisons, there is potential for contributions to discipline-based 
theory and method. 

A second strategy is analysis of professional/alumnae abilities, where 
we have worked to describe ability models of outstanding professionals 
who are not our graduates, and also our graduates, in order to enable 
faculty to define and refine ability definitions, instruction and 
assessment-as-learning, and to evaluate their professional or major 
fields. Because we ask graduates to include examples of activities in 
other areas of their lives in addition to paid employment, we are also 
providing faculty with a picture of abilities that are used in 
personal (e.g., child-rearing; civic involvement; graduate learning) 
as well as professional domains. 

A third strategy builds on the first and second, and extends it for 
more immediate benefits. This is called evaluation of general 
education and evaluation of the major field. As mentioned, we have 
conducted studies that generate ability models for outstanding 
professionals in each of the three largest major field areas: nursing, 
management and teaching (Diez, 1990; DeBack & Mentkowski, 1986; 
Mentkowski, 1988; Mentkowski et al. , 1982). Currently, we are working 
to expand strategies for inter- and intra-individual pattern analyses 
of student performance throughout the major, using data generated from 
faculty-designed external assessment measures, including portfolio 
assessments . 



A fourth strategy is one we call "Teacher as Researcher/Inquirer" 
studies (Research and Evaluation Committee, 1986), which met.ns that 
individual faculty members or groups of faculty conduct research 
projects within or across classes for the purposes of direct 
intervention in teaching and learning activities, so as to improve the 
immediate relationships between instruction and student learning 
(Deahl, 1990; Kramp and Humphreys, 1990). 

A fifth strategy is evaluatitig and validating the 
assessment-as-learning process for individual student development, 
which includes faculty-designed performance assessment measures. We 
have developed a workable definition of contextual validity 
(Mentkowski, 1989; Mentkowski & Rogers; Rogers, 1988) and strategies 
for validating faculty-designed performance assessment measures 
(Alverno College Office of Research and Evaluation/Assessment 
Committee, 1989). The latter have been field-tested with a range of 
colleges and universities in a FIPSE- funded project (Alverno 
College/FIPSE Assessment Project, 1987). 

Finally, as we mentioned earlier, we are working to define criteria 
for "good" assessment (Mentkowski, 1989). Today's symposium is an 
assist for us in this work. 



We are drawing examples today from the first three strategies: 
longitudinal analysis of change related to the curriculum, analysis of 
professional abilities, and evaluation of the major field (refer to 
column 5 in Figure la, "Five Examples Presented Today"). 
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There are a number of other examples from which we could draw to 
examine potential contributions to general research and evaluation 
philosophy, principles, and practice. These are included here to 
provide context for the reader who might be interested in some of 
these other examples. All of these examples are a backdrop to those 
we chose today, as we begin to examine the potential of our work for 
general contributions. 

a) examples where we describing particular uses of educational 
research and evaluation, methods and meeting expected 
criteria 



o personnel issues in maintaining longitudinal designs 
(Mertens & Rogers, 1986); longitudinal data bank 
management and procedures (Ben-Ur, 1986) 

o procedures for maximizing student and alumnae 
participation (Mentkowski, 1988b; Mentkowski & Strait, 
1983; Reisetter & Sandoval, 1987) 

b) examples where we are expanding methods 

o quantitative methods for analyzing Human Potential 
Measures data (Mentkowski & Strait, 1983) 

o qualitative methods for coding the Measure of 
Intellectual Development (Knefelkamp, 1974, 1978; 
Widick, 1975) based on the Perry 1970 Scheme 
(Mentkowski, Moeser & Strait, 1983). 

o qualitative methods for collecting and analyzing 
Perspectives Interview data (Deemer, in press; Much, 
1979) 

o qualitative methods for analyzing Behavioral Event 
Interview data (Deback & Mentkowski, 1986; Mentkowski, 
O'Brien, McEachem & Fowler, 1982; Rogers and 
Reisetter, 1989) 

o contextual validity definition (Mentkowski, 1989; 
Mentkowski & Rogers, 1985; Rogers, 1988) 

o strategies for evaluating and validating 
faculty-designed performance assessment (Alverno College 
Office of Research and Evaluation/Assessment Committee; 
1989) . 

These approaches to institutional assessment have been cited in the 
higher education assessment and college outcomes literature and the 
nature of our contributions has been discussed and critiqued (Astin, 
1991; Erwin, 1991; Ewell, 1984, 1985, 1991; Heywood, 1989; 
Pascarella and Terenzini, 19J5; 1991; Terenzini, 1989b). These 
citations suggest that some external audiences find these approaches 
useful for examining or describing more general approaches. 
Participants at Alverno workshops from other institutions provide an 
external test of such contributions. 




We now turn to examples of contributions to discipline-based theory 
and method in student/adult development, learning and abilities (refer 
to column 5 in Figure la, "Five Examples Presented Today"). 
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SYMPOSIUM QUESTION THREE: TO WHAT DEGREE CAN COLLEGE OUTCOMES STUDIES 
CONTRIBUTE NEW KNOWLEDGE TO DISCIPLINE-BASED THEORIES AND METHODS IN 
STUDENT/ADULT DEVELOPMENT, LEARNING AND ABILITIES, WHICH CAN FORM A 
PARTIAL BASE FOR UNDERGRADUATE EDUCATIONAL PRACTICE? 

Alverno Question Three: Do Alverno Research Methods and Results 
Contribute To Discipline-Based Theory and Methods in Student/Adult 
Development, Learning and Abilities? 

There are a number of examples from our ongoing work that can serve as 
examples of contributions. We list several below, and then move on to 
the selection of the five we chose for this symposium, because they 
illustrate diverse theoretical frameworks in development, learning and 
abilities. These theoretical frameworks, which form the "content" of 
the five examples that follow, are enumerated and illustrated in the 
context of the five examples. 

Figure 3 illustrates the way in which research activities that take 
place in a particular setting with a particular population (see circle 
on the left side of the graphic) benefit from theoretical frameworks 
that are externally derived, but symbiotic to the educational 
frameworks derived from faculty practice (upper left hand box) . 
Similarly, on the right side of the graphic, a "picture of human 
potential" is one goal of discipline-based theory and method in 
student/adult development, learning and abilities. This picture of 
human potential, via discipline-based theory and methods, benefits 
from research activities that are conducted in various settings. In 
either case, contributions from discipline-based theory and methods to 
a particular setting, and likewise contributions frotr a particular 
setting to discipline-based theory and methods, need to be explored. 

How is Contribution Defined and What is Evidence: 
Examples for Potential Methods and Results Contributions to 
Student/Adult Development, Learning and Abilities 

How is contribution defined and what is evidence? Examples follow: 

a) examples where we are studying adult development, abilities 
and learning with accepted strategies and meeting expected 
criteria 

o contributing to the validity of the Human Potential 
Measures (a battery of twelve externally-designed 
measures of cognitive, moral and ego development, 
learning styles and generic abilities) by expanding the 
sample of adult women, providing longitudinal 
trajectories (four data points) during college and 
afterward, and relating change to performance in the 
curriculum (Ben-Ur, Rogers, Reisetter & Mentkowski, 
1987; Mentkowski & Strait, 1963) 
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b) examples where we are expanding discipline-based methods in 
adult development, adult abilities and learning 

o contributing to scoring procedures and manual 
development for the Washington University Sentence 
Completion Test, which is based on Jane Loevinger's 
(1976) theory of ego development (Mentkowski, Miller, 
Davies, Monroe & Fopovic, 1981) 

o identifying indicators of career development for women 
(Giencke-Holl, Mentkowski, Much, Martens, & Rogers, 
1985) 

c) examples where we are developing methods for 
discipline-based theory in adult development, adult 
abilities and learning 

o analysis of intra and inter-individual patterns in 
abilities, learning, personal and professional 
development (Mentkowski, 1990b) 

o measurement of the Perry (1970) scheme of intellectual 
and ethical development (Mentkowski, Moeser & Strait, 
1983) 

o methods for analyzing longitudinal, indepth, 
confidential interviews of student perspectives during 
college and afterward (Deemer, in press; Much, 1979) 

o measurement of employment and career trajectories of 
women 

o Behavioral Event Interview methods for alumnae 
performance studies drawn from David McClelland and 
George Klemp's Job Competence Assessment methods (Rogers 
& Reisetter, 1989) 

o strategies for establishing scoring reliability for the 
Picture Story Exercise (Winter, McClelland & Stewart, 
1981) 

d) examples where we are developing discipline-based theory in 
adult development, adult abilities and adult learning 

o longitudinal descriptions of self-sustained learning 
(Mentkowski, 1988a; Much & Mentkowski, 1984) and other 
developmental domains; description of a H recycling M 
phenomenon in development (Mentkowski, 1988a) 

o descriptions of alumnae generic abilities across a range 
of settings and activities 

o descriptions of professional abilities (DeBack & 
Mentkowski, 1986; Diez, 1990; Mentkowski, etal., 1982) 
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Figure 5 lists some research questions that define 
student/alumna/professional comparisons, and graphs these comparisons. 

These broad questions are: 

o What development, learning and ability outcomes are evident 
during (student outcomes) and after (alumna outcomes) Alverno? 

o How do alumna outcomes compare to student outcomes? 

o How are outcomes developed? 

o How are student and alumna outcomes related to an Alverno 
education? 

o What factors after college help/hinder abilities, learning and 
development? 

o Are student/alumna outcomes "good" compared to 
criteria/standards drawn from Alverno frameworks and 
professional criteria? 
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Figure 5 . Research questions that define student/alumna/professional comparisons. 
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ERIC 



Today, five questions from our research and evaluation program 
are embedded in five examples from our work: 

(1) What broad patterns describe development during college and 
afterward, and how are these patterns related to curriculum 
(cognitive, moral and ego development)? 

(2) What learning outcomes describe development, and what 
curricular elements cause it from the student's perspective 
(self -sustained learning and development)? 

(3) Do women graduates realize their goals (career trajectories of 
women)? 

(4) How do graduates perform in personal and professional domains 
(alumnae generic abilities)? 

(5) How do abilities derived from studies of outstanding 
professionals link to evaluation of student outcomes in the 
major field (professional abilities and student outcomes)? 

(See Figure la, side 2: "Five Alverno Examples" and below.) 

EXAMPLE METHOD 

Cognitive, moral Longitudinal analysis of 

and ego development change in relation to 

trajectories the curriculum 

Self-sustained Same as above 

learning & 
development 

Career trajectories Same as above 

of women 

Alumnae generic Analysis of alumnae abilities 

abilities 

Professional abilities Analysis of professional abilities 

and student outcomes Evaluation of general education 

and the major field 

Three questions organize the five examples we now present for the 
purposes of this symposium: 

o What did we learn that contributes to pictures of human 
potential in development, learning and abilities? 

o How does this study simultaneously contribute to both internal 
and external purposes? 

o What are the unresolved issues? 

A final question, "Which criteria apply?," will be addressed in the 
final section. 
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THE CONTEXT FOR THE EXAMPLES: Setting, Population and Samples 

Figure 3 shows a circle on the left side of the graphic that 
illustrates how research, evaluation and measurement activities work 
interactively with curriculum development and practice in 
interdependent ways. Clearly, in a particular institution, such 
activities revolve around institutional characteristics and the 
population of students who attend the college. For examples of 
studies that we are presenting today, samples are drawn from the 
larger population, over time. Following is a brief description of the 
setting, population, and samples on which results discussed are based. 

Description of the Setting 

Alvemo College, a liberal arts college for women in Milwaukee, 
Wisconsin, has an enrollment of over 2,400 degree students in both 
weekday and weekend time frames. Generally, students are from 
southeastern Wisconsin, are first-generation college students, and 
work during and after college. Alvemo, which has focused for a 
century on preparing women for professional careers, formally adopted 
an outcome-centered approach to its curriculum in 1973, accrediting 
students for progressive demonstration of certain broad abilities via 
the disciplines across all subject areas: communication, analysis, 
problem solving, valuing , social interaction, taking responsibility 
for the global environment, effective citizenship, and aesthetic 
responsiveness. 

Description of \ >e Population 
Characteristics of the student body 

Alvemo' s enrollment has tripled since the beginning of the 
institutionalization of the Office of Research and Evaluation, and the 
start of itB longitudinal study in 1976. In the last decade, the 
college's population of more traditional-aged students in its weekday 
time frame has more than doubled. The current enrollment is 
characterized by a considerable diversity of ages, ethnic and social 
background. 

In comparison to national statistics on post-secondary enrollment, 
where approximately 30% of the women students are over age 30, 
over 50% of Alvemo' s students are over 30. 

Col 1 «ne enrollments for minority students have increased 
na , t lly to about 21%; at Alvemo, the distribution of minority 
students has doubled in the last decade and is now at 19%. Total 
minority enrollment for the State of Wisconsin is 6%. 

About 30% of the students are married. 

A fuller description of the student enrollment and a comparison with 
national statistics is provided in Appendix A. 
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Description of the Sample 
Description of the longitudinal sample 

The longitudinal sample is composed of all women entering the college 
in Fall, 1976 and 1977 (N - 705). (Cress-sectional studies involved 
additional groups but results are not part of the five examples 
presented here.) The rules for eligibility and the rates of 
participation result in certain changes in the sample across four 
times of assessment. At the fourth time of assessment, 358 women were 
eligible members of the sample. 

The sample of 358 is divided into three cohorts: 

- 25% from Weekday students entering in 1976. 

- 32% from the Weekday students entering in 1977. 

- 43% from the Weekend students entering in 1977. 

In comparison to the current enrollment, the longitudinal sample 
tended to be younger, with fewer minorities. Specific characteristics 
of the sample and a fuller comparison with current enrollment in 
provided in Appendix B. 

Issues Related to the Sample 

There are several issues surrounding development of the sample for 
such studies. Usually, in conducting research, one selects a sample 
that enables one to investigate a particular problem. In educational 
research in a particular setting, the sample has been chosen through 
the admissions process. Thus, unknown factors may be at work in 
sample selection; these will likely remain uncontrolled. Hard-money 
institutional funds for research are for the most part, limited to 
those individuals who are in attendance at the college, or who are 
graduates. 

While it may seem at first blush that prior selection of the sample is 
uncommon, we have only to look at the overwhelming participation of 
the college sophomore in social science research to know that such 
selection of the sample prior to the selection of the problem is not 
uncommon in the literature. What is different, however, is that the 
faculty at large, not limited to a committee for the protection of 
human subjects, is concerned with what happens to students as a result 
of participation in research. In our case, the ultimate protection 
derives from the purposes of the research, for improved teaching and 
learning. 

Is the population and the sample adequate? In our experience, the 
question of sample representativeness and generalizability is of equal 
interest to our college faculty and to outside parties. Faculty are 
interested because they, like outside parties, do not want to 
over-generalize from a sample to the current population of students, 
or from group results to an individual student. Thus, challenges to 
sample representativeness come from our own faculty as well as outside 
parties. 

In the beginning, we took for granted that our own faculty would 
generalize from one student population to another because we had 
researched their own students as a whole. The currency of the student 
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population became central to the issue of generalizability, however, 
because faculty were unwilling to generalize just because the sample 
was made up of individuals who had been their own students or were 
graduates. Their concern for dealing with individual differences and 
the individual student rightly surfaced. Thus, description of the 
population and the representativeness of the sample are critical to 
examining issues of generalizability and utilization to Alverno 
faculty, as they arc to outside audiences (Ben-Ur, et al., 1989). 

The questions concerning adequacy of the sample, differ, however. 
Alverno faculty ask: Is the sample representative of the population 
of current students? Is the sample credible? Is the sample 
inclusive? To what extent can one generalize from results from prior 
students or graduates to students now, or future graduates? Outside 
audiences ask: How is the Alverno population representative of 
college students? How is the Alverno population different from 
students at my college? The point here is that questions of sample 
representation and generalizability are equally important for internal 
and external purposes. 
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FIVE EXAMPLES OF POTENTIAL CONTRIBUTIONS TO THEORY AND METHOD 
IN ADULT DEVELOPMENT, LEARNING AND ABILITIES 

Six members of the Office of Research and Evaluation staff will 
illustrate five examples of contributions from research and evaluation 
studies. Each will briefly describe contributions of methods and 
results, and then identify internal and external contributions, that 
is, strengths and weaknesses for providing a) usable findings for 
faculty engaged in improving in-house student learning and programs, 
and b) insights for external audiences engaged in creating new methods 
or theory building in adult abilities, learning and development. 

Synopsis 

Data sources . Results are reported from a) curriculum-embedded 
performance assessments, and b) a longitudinal design using 17 
external measures administered on four occasions (76/77; 78/79; 
80/81; 86/87) to the entire entering classes of 1976 and 1977 N=706) . 
Measures of abilities, learning styles, motivation, cognitive, moral 
and ego development were employed along with indepth, confidential 
interviews, surveys of student perceptions and background 
characteristics, and behavioral event interviews (McClelland, 1978) of 
alumnae. The latter serves as a criterion measure for alumnae 
performance across professions. Student participation rates ranged 
from 84 to 99 percent; alumnae (N«358) rates ranged from 59 to 88 
percent (see Appendix B). Data from curriculum-embedded performance 
assessments in the curriculum, with background factors controlled, 
were related to changes on external measures using multiple linear 
regression, ANOVA for repeated measures and path analysis. Interviews 
were coded via ethnographic and thematic analysis. Student portfolios 
and other curriculum performance assessments were judged on dimensions 
of performance by expert judges and related to abilities that define 
the major. 
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EXAMPLE I: COGNITIVE, MORAL AND EGO DEVELOPMENT TRAJECTORIES 

Synopsis 



Judy Reisetter describes patterns of development on measures of 
cognitive, moral, and ego development in relation to performance in 
the curriculum. These findings contribute to research on these 
developmental domains by extending the longitudinal data we have on 
development of women during college and then into and beyond the 
transition after college. The presence of substantial numbers of 
non-traditional aged wor.ten in the sample who tend to be older (age at 
entrance to college is 17 - 55, x ■ 27) allows examination of the 
robustness of the findings for women across age cohorts. 

Although cross-sectional findings show ego-development gains during 
college, longitudinal findings are consistent with other studies using 
Loevinger and Wessler's (1970) Sentence Completion Test (SOT) that 
show no gains during college (Jane Loevinger, Personal Communication, 
July 22, 1990). Contrary to expectation, data show gains after 
college. Gains accrue during and after college on Watson and Glaser ' s 
(1964) Critical Thinking Appraisal (CTA). Gains accrue during college 
and plateau after college on Rest's (1979a) Defining issues Test 
(DIT) . College gains are associated with progress in the curriculum. 

Findings suggest that college outcomes are broad, developmental, and 
enduring. Students in an ability-based curriculum show gains during 
and after college on a traditional measure of critical thinking. 
Results confirm Rest's (1986) findings on the effects of education on 
moral development, but provide more specific evidence for curricular 
effects. Gains in ego development after college is a new and 
unexpected finding. Developmental trajectories have the potertial to 
inform faculty practice at a broad level. 
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COGNITIVE, MORAL, AND EGO DEVELOPMENT TRAJECTORIES 

What are the patterns of development on measures of cognitive, moral, 
and ego development in relation to performance in the curriculum? How 
do we contribute? In some ways the criteria converge for internal 
benefit and external benefit; that is, contribution of these findings 
on campus converge with contribution to disciplinary research. 
Because of this convergence, college outcomes studies may in some ways 
facilitate systematic collection of data that make analyses of 
important research questions more feasible. 

Let's begin by noting the commonalities of contribution on campus and 
contribution to disciplinary research theory. First, an articulated 
understanding of the effects of different population parameters is 
needed both for use on campus and for understanding the 
general izability of theory across population and setting. In other 
words, a campus question directed toward the research findings is, are 
the findings robust across groups in the sample? A discipline-based 
research question directed toward the findings is, how well does the 
theory generalize to and across the population in this setting? Thus, 
colleges using measures developed in the disciplines (in this case 
cognitive, moral, and ego development) can provide a different setting 
and population to test disciplinary theory, and thereby contribute to 
theory development. 

A second commonality is investigating change through time. 
Understanding change through time is crucial criterion by which 
faculty understand student development and is a crucial component of 
developmental theory. 

A third commonality is showing a relationship between education and 
development. Linking the college's curriculum to student development 
generates faculty interest in the findings. Indeed, most colleges are 
interested in contributing to student development on broad outcomes 
such as cognitive, moral, and socio-emotional development. Through 
institutional assessment, colleges can conduct college outcomes 
studies that include measures of these constructs. This can provide 
theorists with the opportunity to summarize studies of the effects of 
different curriculums on college outcomes or the effect of college in 
general. Colleges are already invested in collecting data on 
individual students and their progress in the curriculum. As they 
become committed to linking curriculum to broad student outcomes, the 
potential for contribution to developmental theory can build upon the 
college's ongoing archiving of data. 

A fourth common contribution is the use of multiple measures that span 
domains. We have noted, along with Peter Ewell, Robert Pace, Sandy 
Astin, and Trudy Banta, that effective use of findings on campus from 
a broad outcome measure is facilitated by having multiple measures. 
Of course, having multiple measures across domains contributes to 
interpreting research findings, as well. On our own campus, the 
multiplicity of measures across domains has been an Important feature 
in the display and interpretation of findings. So, in the alumnae 
follow-up to the Alverno Longitudinal Study, we asked participants to 
complete ten of the externally developed measures of human potential 
that they completed as students. In this paper, we discuss three that 
we have scored and analyzed for the alumnae extension of the 
longitudinal study, in order to illustrate the research contribution 
that accrues from having multiple instruments across domains. 
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Cognitive Development; Critical Thinking Appraisal 

One of the three instruments is the Critical Thinking 
Appraisal (Watson & Glaser, 1964). This is a traditional and 
time tested recognition task, often used as a measure of 
college outcomes. It attempts to measure several components 
of critical thinking. Of the five subscales, we administered 
three of them: the ability to draw reasonable inferences, the 
ability to recognize assumptions of an argument, and the 
ability to draw valid deductions. These subscales may be 
overlapping facets of critical thinking, however (Watson & 
Glaser, 1964) . We used this measure because in 1976 it was 
regarded as a common test of critical thinking. Actually, of 
all of our measures, this measure may map on the least well to 
Alverno faculty descriptions of critical thinking, because of 
its recognition format. 

Moral Development; Defining Issues Test 

A second instrument is the Defining Issues Test (Rest, 1979a; 
1979b; 1986). This instrument is derived from Kohlberg's 
theory of moral development. Rest's conceptualization of the 
development of moral judgment is different from Kohlberg's in 
some ways, for example Rest (1979b, 1986) believes in a more 
complex model that presumes stage mixture, rather than a hard 
stage model. The DIT provides a measure of an individual's 
sophistication in moral judgment through a task that asks the 
participant to choose between alternative considerations that 
should go into making a judgment. The P percent measure that 
we are reporting is interpreted as the relative importance of 
a subject gives to principled moral considerations in making 
decisions in relation to six moral dilemmas. We are reporting 
scores for those participants who completed the DIT in such a 
way that they meet Rest's criteria that increase the validity 
of their score (Rest, 1979b) . 

Ego Development: Washington University Sentence Completion 
Test 

A third instrument is Loevinger's (Loevinger & Wessler, 1970; 
Loevinger, Redmore, & Wessler, 1970) Sentence Completion Test 
of Ego Development. This is a production task that attempts 
to elicit an individual's stage of Ego Development. Ego is 
defined by Loevinger as one's style of life, the unity of 
personality individuality, the method of facing problems, 
opinions about oneself and the problems of life, and the whole 
attitude for making choices in all of life's spheres 
(Loevinger, 1976; Loevinger & Knoll, 1983). For the purpose 
of this presentation, we are using the automatic scoring rules 
for summarizing an individual's total protocol of 36 
responses . 

Having noted these commonalities, let me return to these points in 
relation to our own longitudinal study. In relation to developmental 
theory's interest in change through time, the Alverno Longitudinal 
Study extends the longitudinal assessments over a ten-year period, 
from entrance to college to five years after college. This 
significantly extends the amount of longitudinal information on 
developmental trajectories during college to after college. The 
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transition from college to post-college settings represents a 
potentially important developmental phase. 

In addition, the presence of substantial numbers of returning women in 
the Alverno sample allowed us to examine the robustness of the 
findings across different cohorts of women who range in age from 17 to 
56 years (M = 27) and who have had different experiences. 
Disciplinary theorists are interested in generalizing theories to a 
range of populations and settings, as wall as describing theory 
boundaries. 

Finally, our use of multiple measures that span across developmental 
domains has increased the interpretability of the aggregate findings 
and has provided information relevant to the construct validity cf the 
measures used. In order to demonstrate this point, let us begin with 
a schematic overview of the aggregate Multivariate Analysis of 
Variance (MANOVA) results (See Figure 1-1). The repeated measures 
MANOVA analyses for unequal N's tested for the linear, quadratic, and 
(where four times of assessment were analyzed) cubic trends through 
Time, using orthogonal polynomial contrasts. Weights were used to 
adjust for the unequal lengths of time between intervals. 

As you can see in the Figure 1-1 schematic, we have different trends 
through time for each of the instruments. The differential pattern of 
results on these different measures has implications for the construct 
validity of the instruments. For example, one issue in Loevinger's 
Sentence Completion Test of Ego Development is whether it measures 
something separate from Cognitive Development or Moral Development. 
Loevinger (1979; 1985) points out that interpreting the validity of 
the Sentence Completion Test requires addressing the stage of 
ego-development that is measured. Measures of multiple constructs 
through time allows us to understand how change on one developmental 
measure is related to status or change on another developmental 
measure (cf . Lee & Snarey, 1988; Kitchner, King, Davison, Parker, & 
Woods, 1984). Our preliminary analysis of these aggregate findings is 
not yet related to specific scale values of change, but Lee and Snarey 
(1988) have included data from our study in a meta-analysis of the 
relationship between Loevinger's scoring of Ego Development and 
Kohlberg's scoring of Moral Development. 
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Figure 1-1 . Schematic representation of results on human potential measures 
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In our own analyses, we note that the aggregate tre d for the 
percentage of Principled Thinking levels off during the same period 
that gains are being made on Stage of Ego Development. Given findings 
from other studies, we would not interpret the flattening out of the 
Principled Thinking trajectory as a ceiling effect (cf . Thoma & 
Davison, 1983). Investigating these differential patterns is 
important to the construct validity of these measures because 
interpreting correlation between these measures of related but 
different constructs is difficult (Loevinger, 1979; Rest, 1979a). For 
example correlations can be attenuated by restricted range or be the 
result or the actual covariance of the different constructs. In our 
sample the empirical correlation between the SOT and the DIT at 
entrance to Alverno ranged from .04 to .35. 

Let us now briefly look at change on each of these measures 
separately according to their own scale of measurement. Appendix C 
displays the results of the MANOVA analyses, as well as means and 
standard deviations for statistically significant effects. 

We are often asked how our students perform on traditional measures of 
cognition, because we have an ability-based curriculum. In general, 
participants showed gains during and after college on the three 
subscales of the Critical Thinking Appraisal (see Figures 1-2, 1-3, 
and 1-4), and these gains are robust across age groups. 

Within the domain of moral development, we are investigating change 
through the Defining Issues Test (Rest, 1979a) . The Moral Judgment 
Interview (Colby, Kbhlberg, and collaborators, 1987) was administered 
to a sub-sample, but we have not analyzed the alumnae data yet. 
Again, you can see that aggregate change on the Defining Issues Test 
is toward greater preference for Principled Thinking about moral 
dilemmas during college, and that these gains plateau out after 
Alverno (see Figure 1-5) . We have found that the gains during Alverno 
are associated with our primary measure of progress in the curriculum, 
and that this relationship to progress in the curriculum remains even 
after we have used regression procedures to statistically control for 
background variables (see Mentkowskl and Strait, 1983). These results 
are consistent with Rest's (1979b; 1986) finding of a robust 
relationship between formal education and the sophistication of moral 
judgment. We have demonstrated a relationship to a coherent, 
developmental, ability-based curriculum. 

In contrast, we did not find a robust and positive relationship 
between student progress through the Alverno ability-based curriculum 
and change on the Critical Thinking Appraisal (see Mentkowskl & 
Strait, 1983). Findings for both the Defining Issues Test and the 
Critical Thinking Appraisal are consistent with the expectation that 
college outcomes are broad, developmental, and enduring. The 
relationship of the Critical Thinking Appraisal to progress in the 
curriculum remains undemonstrated. The lack of robust relationship to 
the curriculum might be attributable to measurement issues (see Rogers 
& Mentkowskl, 1985). For example, one issue Involves the sensitivity 
of the progress in the curriculum measure in relation to the changes 
on the Critical Thinking Appraisal that occur across widely spaced 
Intervals of one and a half to two years. 
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Fi gure 1-2 . Critical Thinking Appraisal: Inference subscale. 
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Fi gure 1-3 . Critical Thinking Appraisal: Recognition of Assumptions 
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Fi gure 1-4 . Critical Thinking Appraisal: Deduction subscale. 
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Fi gure 1-5 . Defining Issues Test: Moral Judgment (P%) 



Moral Judgment (P%) 
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Nonetheless, we have been able to develop some articulated 
understanding of the probability of a causal relationship between the 
curriculum and our outcome measures. The primary progress in the 
curriculum measure that we have used capitalizes on the explicit 
sequencing of ability levels in the curriculum. Our reasoning was 
that if the Alverno ability-based curriculum is a causal factor in the 
growth demonstrated on these measures, then the number of 
performance-based assessments successfully completed in the curriculum 
should be associated with this growth. 

Of course, other possible explanations for the association between 
progress in the curriculum and changes on the DIT are always possible, 
but by collecting information on a range of variables that describes 
much of the existing heterogeneity in the population (age, program, 
prior college experience, high school GPA, etc), we have been able to 
investigate — to some degree — whether these variables might qualify 
the findinos. Although we recognize the casual analyses we have 
conducted are merely consistent with our causal inferences, taking the 
step toward linking a specific and whole curriculum with gains on 
human potential measures provides an important interpretative fuxjrum. 

On one measure we did not show any change during college. Although 
our analysis of a cross-sectional comparison group suggested 
ego-development gains during college, longitudinal findings are 
consistent with other studies using Loevinger and Wessler's (1970) 
Sentence Completion Test (SCT) that show no gains during college 
(Loevinger, 1985), although some change is reported by others (Loxley 

6 Whitely, 1986). But, because our sample includes a post-college 
trajectory for these women, wo are able to give information on 
post-college growth, and we are not aware of any other studies with 
information on this post-college transition. Loevinger (Personal 
communication, July 1990) has told us that she does not expect growth 
after college. Our MANOVA analysis of the scores, however (see Tables 

7 & 8 in Appendix C) , suggests that there is upward growth on Ego 
Development after Alverno (see Figure 1-6). One of our first 
questions is, is it real? There is a statistically significant linear 
and quadratic effect (Table 8 in Appendix C) , which supports the 
interpretation of no change during college followed by gains after 
college (see Figure 1-6) . We have not yet analyzed how two conditions 
of test administration (mailed to alumnae versus alumnae coining to 
Alverno) might affect the alumnae results. We have only just begun 
the process of interpreting and analyzing the data. One of our 
questions is how background variables might qualify or help explain 
the results. We do not feel that we have investigated this question 
adequately yet, and the number of cases in the analyses prevents 
matching participants on all background variables. Previous research 
suggests that Ego Development may be positively associated with 
Parent's Socio-Economic status (see Browning, 1987) when the age of 
the participant is controlled. In our sample approximately 25% of the 
women have a mother who has not completed high school. For the 
purposes of the MANOVA, these women were compared with those whose 
mother's completed a high school degree or above. Only the first, 
second, and alumnae times of assessment were included in this 
analysis, yielding minimally adequate cell sizes of (Ns ■ 11, 15, and 
30) for the lower values of Mother's education. 
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Fi gure 1-6 . Sentence Completion Test of Ego Development: 
Automatic total protocol (TPR) rating of 36 items. 

Ego Development Stage (TPR) 
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Although neither Acre at entrance nor Mother's Education were 
statistically significant in this MANOVA analysis (see Tables 10 & 11 
in Appendix C) , these analyses should be taken as preliminary and only 
suggestive. We note again that there is some restriction in the range 
of the Mother's Education variable and that the participant's Age may 
be associated complexly with other variables in this sample. Still, 
for our sample we have not yet uncovered any simple relationship 
between Age, Mother's Education, and change in Ego Development. We 
tentatively conclude that gain in Ego Development is relatively robust 
across the broad Age categories and the high school versus higher 
educational attainment category for Mother's education. 

Contributions. 

The current findings contribute to the existing information on adult 
development, by extending the information researchers have on Ego and 
Moral development during college to the transition to five years after 
college. We have been able to suggest that there may be post-college 
changes in Ego Development, and to confirm that the development of 
sophistication in moral judgment may be generally maintained after 
college. The data is significant not only because of the longitudinal 
time frame it covers, but also because of the multiplicity of measures 
used. Already, we have been able to support the construct validity of 
the measures by showing showing divergent trends on the related, but 
different constructs. As we seek to understand the hows and whys of 
these divergent trends, we expect that we able to contribute to the 
disciplinary understanding of how development across moral, ego, and 
cognitive domains are related to one another. Already, we have 
contributed by linking the investigation of change across these 
measures to the participant's progress in the curriculum. 

Unresolved Issues 

Although there is a meaningful convergence between many of the 
criteria for disciplinary research and for utilization of findings on 
campus, there is also some divergence. For example, one central 
criteria for utilization is the timeliness of reports. Findings from 
the longitudinal administration of the student potential measures were 
reported as soon as possible (Mentkowski & Strait, 1983; Mentkowski, 
Moeser, & Strait, 1983) . The developmental theory used in the 
research has informed thinkincj on curriculum revision. For example, 
Perry's (1970) theory of intellectual and ethical development and 
Kolb's (1984) experiential learning theory have resonated with faculty 
thinking. 

Nonetheless disciplinary researchers are in some ways more invested 
than faculty in using specific findings from the longitudinal measures 
of human potential. In particular, the findings might have a 
half- life on campus, where their initial impact decays over time. 
Recently, the time frame for collecting the longitudinal data has 
begun to lead to the unexpected concern over whether the findings from 
the study generalize to the same college from which the data were 
collected. Faculty are acutely sensitive to the individual 
differences in their students and are constantly adjusting and 
improving the curriculum. 

A new faculty member may be excited about interpreting the data for 
the purpose of curriculum revision. A more experienced faculty member 
may view the data as more likely true of former students only, taught 
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when the curriculum was less developed. Alumnae findings on 
cognitive, moral, and ego development trajectories will, we expect, 
infuse a renewed interest in the findings, as faculty confront the 
question of how enduring the effects of the curriculum may be. 

We are still addressing a range of questions. What measures axe 
appropriate for college outcomes measures, and how much change might 
we expect? Ego Development does not appear to have changed during 
college, but then it is a very broad construct. Did college in some 
way prepare students for development after college? Can we relate 
change during college in moral development to change after college in 
Ego Development? Does performance in the curriculum during college 
relate to development after college? What range of questions do we 
put to the data, and how do we refine these questions for an 
integrated analysis strategy, when multiple staff are conducting 
analyses that raise their own questions? What are the better 
alternatives for investigating intra- individual change and 
inter-individual change? Who changes and why? 

We are still assessing one issue that may be of particular interest to 
others embarking on a longitudinal study of college outcomes. By 
assessing the entire entering class, our population can be described 
as a sample of opportunity in relation to the examination of the 
influence of background variables on development. For example, 
because of the number of returning women in the college, we have a 
good distribution of age in our sample, and sometimes we have used 
this age variable as a theoretically significant surrogate for 
maturation. In our population, however, returning women differ in 
more ways than just age. We are still working on ways to better 
unconfound the variables in our sample, and although we expect some 
success in this, we also expect that there will limitations on how 
well we can unconfound some of these variables. One of our tasks is 
to find better ways of displaying as well as investigating the meaning 
of the background differences in our population. Nonetheless, the 
heterogeneity of the sample, particularly with respect to age, can 
contribute to a life-span view of development. 

In sum, the analysis of the battery of human potential measures 
enables us to paint a broad picture of the development of women across 
ten years, from entrance to college, to five years after college. We 
have linked some of the changes during the student years to the 
curriculum. This broad picture of development increases faculty 
understanding of their students, and provides a framework for how they 
think about their teaching. But, faculty are also interested in more 
specific relationships between their teaching strategies and student 
learning. The broad picture of development provided by the human 
potential measures does not address these specific relationships. The 
next section will discuss longitudinal interviews of these same 
students. These longitudinal interviews have illuminated specific 
teaching and learning outcomes, and the relationship between them. 
So, while the human potential measures are able to show the broad 
effect of the curriculum, the Perspectives Interviews are able to show 
how specific teaching strategies are related to development. 
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Synopsis 

Deborah Deemer presents descriptive patterns of students' perspectives on 
their learning, growth, and development from longitudinal interviews in 
relation to curricular causes as attributed by students. Teaching and 
assessment methods that facilitated learning and personal growth generally 
validated faculty understanding of their practice. These include feedback, 
modeling, instructor attention, self -assessment, practice, and opportunities to 
integrate abilities. The description of students' development in understanding 
the use of criteria in self-assessing performance has resonated with faculty 
experience. Alumnae show more sophisticated learning patterns than students. 
Two-year alumnae have demonstrated different strategies for balancing family, 
work, and educational responsibilities, which appear useful for student 
services personnel. Patterns of learning confirm Kolb's (1984) theory of 
experiential learning. Developing ethnographic and other qualitative 
methods to analyze student perspectives interview texts is challenging, 
but likely to have long-range benefits for understanding how students 
experience curriculum, internalize and act out of acquired outcomes 
during and after college. 
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SELF-SUSTAINED LEARNING AND DEVELOPMENT 

This section provides an example of longitudinal analysis of change 
studies that demonstrate relationships to the curriculum using student 
and alumnae interviews. It points to contributions already realized 
as well as potential contributions (Figure II-l). Realizing future 
contributions will challenge us to grapple with a number of unresolved 
issues. 

Contributions 

The use of confidential, indepth, longitudinal interviews of students 
and alumnae contributed to validation of the curriculum, provided 
feedback on faculty practice, and was used to develop a student 
learning inventory that gives beginning students feedback about their 
strengths and weaknesses as learners at Alverno and about faculty 
expectations. These benefits, which speak to the value of conducting 
interviews, is a finding that could be useful to others interested in 
studying student and alumnae outcomes. How faculty and Office of 
Research and Evaluation staff collaborated in the use of findings from 
the interviews might also contribute to the thinking of others working 
in institutional assessment. 

Findings about aspects of faculty practice that facilitate self 
sustained learning and development contribute to the knowledge base of 
educational research. Domains of meaning (see Figure II-2) we are 
using in an analysis of the alumnae interviews hold the promise of 
future contributions. Aspects of the curriculum and faculty practice 
that facilitate growth in human potential will provide an additional 
contribution to the adult development literature. The evolution of a 
method to analyze the alumnae interviews and revisit interviews from 
the college years might stimulate the thinking of other researchers in 
college outcomes assessment (Deemer and Mentkowski, 1990) . 

Sample 

Between 1976 and 1981 a random selection of students from the entering 
classes of 1976 and 1977 were asked to talk about their educational 
experiences at Alverno at the end of each college year. These same 
individuals were interviewed as five year alumnae. Longitudinal 
interviews are available from 99 students; 71 participated as alumnae. 
There are 73 participants who completed at least four interviews; 51 
completed all five interviews. The interview was also conducted with 
a cross-sectional sample of 32 alumnae from the 1978 graduating class. 

The interview method 

The primary purpose of the Perspectives Interviews is to give our 
students and alumnae at. • y, ~oortunity to speak for themselves about 
their educational and p c-college experiences. The interviews 
provide a window into how the student is constructing the learning 
process, responding to it, and to what she attributes her own 
development and learning. Faculty can then make a judgment about 
whether the student's construction of the curriculum is what faculty 
intend it to be, whether faculty and student constructions become more 
or less attuned over time, whether a student is developing a surface 
or indepth understanding, and what facilitates emergent common, 
unique, and internalized understandings. 
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Figure il-l . Understanding student and alumnae development and contributing 
to college outcomes research and theory through student and alumnae 
perspectives interviews (Pi's). 
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Figure I 1-2 . Projected domains of analysis of the student/t Tumna 
perspectives interview approach. 
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Each Interview provides an independent, external judgment regarding 
the curriculum' s validity and offers information about what , from the 
student's perspective, is working in the curriculum and what could be 
improved. How the students' construction changes over time provides 
evidence about the outcomes of college. 

The alumnae interviews provide information about how these 
constructions are transformed after college. They also give us 
insight into the settings and life events our students confront after 
college; what life has demanded of them and whether college learning 
and abilities have been useful in helping them meet these demands. In 
the last two weeks we learned that ego development changes after 
college. It will be fascinating to examine what contributed to this 
change, and for whom. Can we link changes in college on some other 
domains to changes after college on ego development? 

Findings from interviews during the college years 

The focus here is on findings from an analysis of interviews conducted 
at four points, at the end of each year in college. The results of 
the interview analysis are rather extensive. A fuller description of 
the findings for the first four data points are provided in Much and 
Mentkowski (1984). For our current purpose a few broad findings from 
their work are highlighted with an example of how these findings were 
infused into Alverno practice. 

Much and Mentkowski report that students differed in how well they 
grasped the overall purpose of the curriculum and it's relevance to 
their personal and professional goals. This difference was related to 
a more general ability to reason, infer and make relationships between 
the abilities taught toward by the faculty and how students used them 
in and outside class, on and off campus. The number of students who 
constructed relationships among abilities taught by the college and 
their performance in both personal and professional aspects of their 
lives increased over the college years. More advanced students had a 
better understanding of how their learning experiences made sense in 
terms of what the college and the student were working to achieve. 

Student constructions of the learning process reveal a developing 
understanding of the role of criteria ( which are behavioral 
descriptions of performance students and faculty use in assessment of 
individual work) (see Appendix C) . Beginning students were apt to 
construct criteria as vague and arbitrary standards beyond their 
control. Hie criteria were understood as too explicit or "too picky" 
directions for how much content to learn to "pass" . As they 
progressed students saw criteria as pictures of the abilities to be 
performed. Later they saw criteria as open to interpretation and 
providing a framework for feedback and self -assessment (see Appendix 
D). 
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The ability to use criteria to evaluate their own performance, i.e. , 
to self -assess, plavs a central role in the student's ability to 
engage in indepenc. c learning and development after college 
(Mentkowski, Much & Giencke-Hall, 1983). Students became increasingly 
capable as independent learners as they worked within the curriculum 
(Figure II-3). 

Also during college, students developed a better sense of how they 
learned, and how they thought about learning. At the same time they 
became aware of and began to use diverse ways of learning. 

Of particular interest to the faculty, students identified aspects of 
faculty practice that helped them in these three broad areas of their 
learning: making relationships, taking responsibility for their 
learning, and become increasingly capable in a multiplicity of 
learning approaches (Figure II-4) . Excerpts that illustrate the 
evolution of students' thinking on these domains are provided on 
Figure II-5. 

Utilization of the findings 

Faculty interpretations of these findings played a role in the 
development of a learning inventory used in the orientation of first 
year students, i.e., The Student as Learner Inventory. The history of 
the development of this instrument and both faculty and student 
response to its use is described by Rogers (1988). 

In brief, Assessment Committee members and Office of Research end 
Evaluation researchers used student constructions gleaned from the 
interviews in the creation of the inventory. The whole faculty had 
spent time thinking out loud about characteristics demonstrated by 
advanced students and were working on a set of characteristics to 
depict beginning and developing students. This data was brought to 
bear to create a picture of faculty constructions. The combination of 
student and faculty constructions shaped the inventory. 

While faculty historically meet at particular moments in time to move 
forward certain aspects of their work, they bring to their 
conversations understandings of their students drawn from a myriad of 
interactions and sources. Development of the student as Learner 
Inventory was no exception in this regard. As faculty were formally 
and informally gathering to develop pictures of their students' 
evolving understanding as learners, interview analyses brought to the 
conversations, findings about changes in student constructions 
observed in the interviews with students over their four years in 
college . 

Out of this team work, research staff developed rating forms that 
integrated what faculty had already articulated with statements 
generated out of findings from student interviews. Faculty used these 
forms and provided feedback not only about technical aspects of the 
scales but also ensured that their educational philosophy was infused 
in the instrument's design (e.g., persisting until the forms were 
written in a way that would make it possible for them to provide 
feedback to students) . The product of this collaborative work is the 
current version of the Student as Learner Inventory which is used to 
0 orient students and which is still revised based upon practice. 
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Figure I I- 3 . Self -sustained learning in students. 
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Figure 11*5. Student perspectives on their learning during college: Excerpts from beginning, developing and 
advanced students. 
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... the analysis and problem solving, I've 
been doing all my life ... But you're not 
really aware of all the things you do, and 
once you are aware of each part I think 
your skill is sharpened and more easily 
used and applicable to different situations. 
CLU's that compare, like science and ait, 
your Arts and Humanities CLU, I think 
that was really important because I don't 
think I really looked si all the different 
aspects of art and tried to compare my 
values to someone else's values. Even 
though we're in the same society, our 
values are different and I don't think I 
really was awaitt of that before. 



ADVANCED 

They've asked me to see things in creative 
ways and to look at something differently 
and I wasn't able to do that very much 
before. I guess I kind of fell into a pattern 
and saw things one way or another way ... 
So it has caused me to take things and see 
them from many different point of view 
and that's challenging because it's much 
easier to say I'm a psychology minor and 
I'm interested in psych and if I'm going to 
look at a person's values I'm going to look 
at what's in their head. To try to get 
values out of a hiochem experiment or 
something, that's challenging. So looking 
for relationships in a lot of things and 
looking for universality where there seems 
to be none is really hard on your head, it 
really is. 
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ROLE OF CRITERIA 



BEGINNING 

I like situations where I am told exactly 
what to do. Hiit'i another thing I had a 
hard time getting used to here, was that I'd 
say how long should this be, and they'd go 
well as long as you fed is necessary. Well 
what's that, a paragraph or a page? They 
give you criteria for what you have 10 do 
but you're never sure if you are meeting it 
all. I live in fear of leaving one out 



DEVELOPING 

Like your valuing, your social interaction 
all this, they have general rules and first 
you spot them somewhere and then you 
have to apply them and people are 
watching you. You know, you're asseiied, 
it's like you have to do it. Well after 
awhile I've incorporated many of the 
things into my own life ... you were forced 
to do it and now you're kind of doing it 
automatically ... like now I have critical 
incidents ... and it's like you make your 
own general rule but look out to see if you 
are doing valuing, are you doing social 
interaction, arc you doing problem solving. 
So then because of that I have the rales 
internalized ... (W)hen it happens in a 
situation I can think, oh yeah, I did it 
And then I go home and sit and write out 
what I did, what Alvemo says you should 
do and then compare it. 



ADVANCED 

... my change in learning was probably 
from a memorized approach to an 
application approach. And I think that to 
an extent the, looking at the learning 
process it's shifted in a similar manner. 
Like from the letter of a law to the spirit of 
the law kind of approach. Now that's land 
of vague but I think in the lower level 
competencies ... what I did for each project 
was probably very dictated and in a sense 
bounded by what the criteria said. You 
know, the criteria says you have to do this 
... it give* you specifically what you have 
to do and I did it word for word, you 
know. It says now compare, so I 
compared. It sort of structures what it is 
you're doing. Now that I would kind of 
parallel with the memorized kind of 
approach. As I move into the upper levels 
I think that I was aware of more flexibility 
within those criteria outlines. For example, 
in an upper level project that I completed 
this semester, I was suppose to ... do ... a 
comparative analysis and then compose a 
code that would ... take into account 
research in a given area. Like how should 
researchers in a specific area relate to their 
clients and to the public and all that and 
compose a code that dictated that Well 
after doing the comparison ... my analysis 
led me to the conclusion that a new code 
wasn't necessary ... (A)ll that ... researches 
in that area needed, was to apply the code 
that existed ... and that was my statement 
and I didn't compose my code as I 
probably would have, you know, awhile 
back ... I'm trying to set up a parallel and I 
don't know if it's particularly clear, but I 
guess the basic structure of it is moving 
from that letter of the law kind of approach 
to the spirit of the law kind of approach ... 
I realize that this ... criteria or this 
statement wasn't set in stone and that given 
my conclusion ... I didn't have to write that 
code. That my analysis led me to ... make 
another conclusion ... in going through! the 
process I think you become aware of the 
fact that those criteria are not set in stone, 
that they are flexible. And that the way a 
given criteria was outlined, how to do 
something, is not the only way to do it ... 
and then ... a lot of things open up for you, 
it's a lot broader ... spectrum you can 
pursue in terms of learning. 
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DEVELOPING 

... I always thought college wis, you tit in 
a room and you read a book and if you can 
memorize everything, its pretty good, you 
pass your tests, no problem. Well it takes 
a Utile more than that here. You have to 
use your head a little bit more. You have 
to be able to reason and solve problems 
and all that kind of thing, which is more 
than just memorizing. It's harder because I 
can sit there and memorize for a test, and 
memorize and memorize and know it all, 
its all down there, and then when it comes 
to the test I have to be able to take all that 
memorization and apply it to different 
things, and I can't always do that, and to 
relate, sometimes its really difficult 
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WAYS OF LEARNING 

ADVANCED 

... you ought to have a major background 
of all different theories and then when a 
situation arises ... choose from that 
background which one you are going to 
follow, that is what I see this as. I have 
been taught all these different ways of 
being able to learn and being able to 
demonstrate what I have learned and I 
evaluated each one ... and which one I am 
better at and stuff like that When a 
situation does arise I'll be able to say this 
is what I am going to use or this is what I 
need to work on in this situation. 

I'm always looking for ways to do my 
CLU's ... And when I approach a subject I 
look at how am I going to do this CLU? 
And then how is what I am learning in the 
lecture or lab going to aid me in doing this 
or performing this certain behavior which I 
have to be able to do. Writing a paper or 
finding values in scientific articles or what 
ever it would be, I'm always looking for 
ways of how am I going to do this? 
Before it was just studying for a test, but 
now you are constantly reading a book or 
reading an article or listening to a lecture 
to aid you with doing performance. 
Because later on, after I graduate, I'm not 
going to be able to take a test and give it 
to the patient. You have to be able to do 
certain things and just talking or working 
with your fellow employees in your own 
life, there are certain things that you have 
to be able to demonstrate every day. 

So I think I've learned a great deal through 
the other people, teachers, when they are 
helping me specifically or other students, 
when you sit down and talk to them. 
Because you can learn a lot more. You 
can get a lot more perspsctive rather than 
just reading from a book. Because there 
are a million ways to look at something 
and if there is a continuing education 
student in one of your courses ... right now 
... an example comes to mind because we 
just finished with a childhood sequence in 
nursing. And so the women who have had 
children, they can help you too. So it's 
like you've got a teacher, a nurse who is a 
teacher, she can give you experiential 
advice or views, and then you also have a 
lot of students who are mothers themselves 
and you also have students who are sharing 
the common experiences that you are. You 
can share them together and you're getting 
a lot more points of view rather than jus* 
sitting and not talking about what you are 
learning. 
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In response to an open-ended statement added to the inventory, "What 
did I learn about myself as a learner by taking this inventory" (N*80) 
almost all students felt they either gained insight into or reaffirmed 
their view of their strengths and weaknesses as a learner. Most said 
they were prompted to think about independent learning or about using 
feedback more effectively. Many wrote about how they would apply what 
they learned about themselves as a learner to their course work or 
personal relationships. 

U nresolved Issues; An evolving method for analysis of the alumnae 

interviews . 

Faculty perspectives have been useful in the evaluation efforts of the 
office and provide substantive contributions to the understanding of 
adult learning and development. The process by which knowledge is 
arrived at and the character of the findings are somewhat different 
than those derived by researchers working within a particular 
theoretical or discipline based framework. This section addresses the 
experience of one researcher working within the evaluative context of 
Alverno and hopes to illuminate what happens when discipline based 
research gives way to faculty theory in practice. 

The researcher primarily responsible for the analysis of the five year 
alumnae Perspectives Interviews has commitments to developmental 
psychology as a discipline and is eager to begin an emergent analysis 
that would empirically identify associations between growth in human 
potential and experiences, orientations and interests revealed in the 
student and alumnae interviews. Barriers to movement on this project 
have been generated by the need to reconcile both methodological and 
value perspectives. 

A central methodological issue we confronted was how to reconcile 
conflicting ideas about the importance of keeping blind to alternative 
data sources. So for example, one perspective holds that aggregate 
findings across participants completing the same instrument should be 
"independently" derived and analyzed before we empirically search for 
relationships across the data sources. An alternative view is that in 
examining information generated by individuals across the data sets, 
meaningful relationships will emerge. The importance of these 
relationships could not be initiated by researchers working solely 
within one data source. 

A related issue arose from a desire to have staff who were working with 
different data sources begin their initial analyses with information 
from the same participants, i.e., to use the same sample. In this way 
we could work towards an initial pilot study and preliminary 
presentations that would reflect the integration of different strands 
of student development and abilities. Some staff wanted to use a 
random sample so, for example, they could minimize the effects of rater 
drift. Other staff preferred to hand pick an initial sample that would 
ensure a developmental dispersion. Splitting the data into 
"exploratory" and "confirmatory" subsets helped, but did not resolve 
the issue. 

But as we developed procedures that, while cumbersome, would move us 
beyond these methodological issues, it became clear that other less 
articulated concerns would need to be addressed before this project 
FR?C could move forward. More specifically, the college has been engaged in 
an emergent understanding of their practice. While faculty are well 
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read in their fields and draw on discipline based theory, the evolving 
practical understanding of the faculty can't be viewed from the lens of 
any particular extant theory. Analyzing the interviews out of one 
discipline perspective, such as moral judgment research, was thought to 
provide too narrow a focus for the study. The question became, "How do 
you use the understandings of a pluralistic, multicultural, and 
interdisciplinary faculty as a theoretical base in the analysis of 
interview texts?" 

In our current plans for the Alumnae Perspectives Interview analysis we 
opted to have the Director of the Office of Research and Evaluation 
(ORE) provide a starter set of domains that integrate current faculty 
interests and prior research findings (see Figure II-2). The Director 
will participate in early interview analyses to help the less 
experienced researcher understand what is relevant to these domains. 
We also expect that new domains will emerge out of the data analysis. 
Feedback on the desirability of conducting an analysis in these domains 
will be provided both by the Director of ORE and faculty on the 
Research and Evaluation Committee. 

It is clear that the value commitments of an institution provide 
boundaries around what is to be studied and helps to establish 
priorities. When we identify student outcomes and aspects of faculty 
practice related to growth in human potential both the college and the 
adult development literature will be enriched. But far more is 
provided in the interviews than is directly tied to faculty practice. 
For example, an analysis of interviews with two year alumnae yielded 
interesting ideas about how women integrate family and careers 
(Mentkowski, 1983). While this work is fascinating and made a 
significant contribution to the literature, it contributes much less to 
specific faculty practice and may provide little in terms of validating 
the curriculum. Faculty interest in understanding student lives goes 
beyond specific irformation about their practice. So we will not want 
to lose sight of information that illustrates how alumnae balance 
involvements in school, work, their families and in the community. 

A method of efficiently gathering and organizing information would be 
helpful. We need a method that responds to questions most engaging of 
the faculty while simultaneously recording where data is available that 
will speak to future questions of interest. The ability to quickly 
pull information together will help us respond to the changing 
interests and needs of faculty. Advanced in computer technology for 
use in qualitative data analysis may be helpful in this regard. 

Preliminary emergent findings will be reviewed by members of the 
college. Faculty and staff will help in our interpretation of what 
information is relevant to the domains and where we will most 
profitably invest our energies. By infusing understandings that 
faculty and staff have gained from their practice, the relevance, 
accuracy, credibility and usefulness of the findings will be enhanced. 
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EXAMPLE IK; CAREER TRAJECTORIES OF WOMEN 

Synopsis 

Tamar Ben-Ur presents data from the questionnaires used in the Alverno 
Longitudinal Study. Colleges often use questionnaire data on students and 
alumnae to evaluate and improve their programs. This example demonstrates how 
questionnaire data can be used to give basic data on indicators of occupational 
attainment. In addition, this example discusses issues that arise around both 
the conceptualization and measurement of the careers of women. In this study, 
women' 6 careers are analyzed for family, educational, and occupational aspects 
of women's careers. 

Analysis of the questionnaire suggests that most alumnae are in professional or 

managerial positions commensurate with a college degree. Some 

non- traditional ly aged women tended to remain, however, in clerical positions, 

suggesting that there may be constraints to career mobility for some 

non- traditionally aged women. Although questionnaire based data from college 

outcomes studies may shed light on women's careers in general, the 

questionnaire based data is most effective in providing internal reports to the 

college. 
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CAREER TRAJECTORIES OP WOMEN 

The Perspectives Interviews and the Human Potential Measures 
contribute to an enlarged understanding of the trajectories of women's 
lives, learning, and development, The institution also has an 
interest in basic data on what alumnae career, family, and educational 
trajectories look like. Are they satisfied? Do they feel well 
prepared by Alverno across different activity areas and abilities? 
How many work? Do they work in a field related to their major? Do 
they experience occupational mobility? Are they currently in a higher 
level than they were when they entered college? Is there 
inter-generational mobility? Are they in a higher level position than 
their parents? 

These questions address basic objectives that many students bring with 
them to college and that are included in the mission of the college: 
The personal and professional development of students. Are alumnae 
developing personally and professionally? So the example that we are 
giving you is showing data that were collected and analyzed to answer 
institutionally driven questions. The data primarily serve an 
institutional reporting function, and are less oriented toward 
contributing externally. 

At the same time, because of the diversity of ages and backgrounds axtf 
the combination of liberal arts and professional majors, the results 
produced in this study have the potential to contribute to the 
knowledge base of adult development and college outcomes while they 
answer specific college questions. 

We now share lessons learned and unresolved issues regarding the 
measurement and the description of career trajectories for women in 
the context of institutional reporting. Employment and family 
background data were collected through a variety of measures in the 
Longitudinal Study. Towards the end of the 4th year at Alverno, which 
for most students is shortly before graduation, the students were 
asked about their paid and unpaid employment before and while at 
Alverno. They were also asked about their expectations relating to 
their first job after college and about their career goals. 
Approximately five years after college, the alumnae were asked to 
indicate their primary activity (which includes paid employment, 
family, volunteer and self -development) . A number of questions probed 
in more detail employment and education since graduation, the 
abilities which alumnae perceived as important in current employment, 
the preparation they received at their college and satisfaction with 
their primary activity. 

Employment and Mobility 

For our five year alumnae, we learned that 95% were currently 
employed. Given that most of the alumnae continue to work, other 
questions followed that help interpret employment characteristics in 
terms of the college's concerns. 

NOTE: For our current analysis, we used the 1980 census-based 
major occupational groupings suggested by the U.S. Department of 
Commerce. We collapsed the 13 major categories into three basic 
categories based upon the mean socio-economic scores for each of 
the 13 categories (Stevens and Cho 1985) . We did not use the 
socio-economic scores assigned to the 1980 census-based 
occupational codes, since existing census job specifications are 
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ineffective in describing the myriad of job titles used by our 
participants, especially in management and management-related 
occupations. They also do not provide directions for coding part 
time jobs. 

o Are Alverno graduates in a higher status positions than their 
parents? To what extent has there been inter-generational 

mobility? 

While some colleges may not ask this question, the mission of Alverno 
includes meeting the diverse needs of women of varied economic 
circumstances. The majority of our students are first generation 
college students and more than half of them come from blue collar 
families. In particular, non-traditional ly aged students tend to come 
from lower socio-economic status. However, since the majority of 
alumnae are in professional or managerial positions, current job 
classification of the participants is not related to their parents' 
job classification. 

o Are they currently in a higher level position than they were upon 
entrance to college? 

This addresses the extent to which non-traditionally aged women 
improved their socio-economic status following graduation. For the 
non-traditionally aged women, there is some preservation in 
occupational category before and after Alverno. Non-traditionally 
aged women may face some barriers to pursuing or making career changes 
after college, while traditionally-aged students seem to be more 
effective in beginning their occupational careers. (85% of the 
traditional age group who are employed at Time 4 vs 76% of the older 
groups are currently in professional/management related jobs . ) It 
appears that some older women stay in clerical jobs. 

o Are Alverno graduates generally employed in positions that are 
commensurate with a college degree? 

None of the graduates is in a blue collar position five years after 
graduation. Seventy eight percent of the participants are employed in 
professional and managerial positions that usually require or at least 
prefer incumbents with a college degree. Only 17% are in sales and 
clerical jobs category that was the modal category for the pre college 
students. 



Perceived Preparation By Alverno And Job Satisfaction 

Perhaps because, Alverno has an ability-based curriculum, faculty and 
staff are particularly interested in the alumnae level of ability and 
satisfaction on the job at least as much as they are interested in 
their mobility. To answer these questions, we identified 25 abilities 
which we believed are essential to efficient performance of a variety 
of jobs and which we believed the college attempted to develop. We 
asked the alumnae how often each ability is required by their job and 
then to indicate how they think their abilities compare to the 
abilities of others who are engaged in the same or similar job title. 
Table III-l reveals that all the abilities we selected are perceived 
by the alumnae as required on their jobs at least sometimes. 
Evidently, some abilities are perceived as required more often than 
other. The table also show that participants tend to perceive their 
abilities as strong or stronger than the abilities of other people in 
the same position. We correlated the 25 pairs of ability variables. 
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All 25 correlations were significant ranging from .15 to .56 (Table 
III -2), supporting the conclusion that our alumnae feel confident and 
competent on their job. 

Our survey showed a very high level of job satisfaction with first job 
after Alvemo. Participants tended to believe that their jobs are 
consistent with their abilities and that they have a chance for 
mobility. However, these levels as reported by alumnae are lower than 
their expectations at graduation, and a small number of alumnae were 
dissatisfied with their first jobs. 

In general, alumnae rated their preparation at Alverno highly. It is 
interesting to note that the college was ranked higher in providing 
"potential for growth," "preparation for higher education" and "life 
long learning" than for preparation "career and economic growth. " 



Analysis A cross Domains of Life Trajectories 

Many of our research questions are raised through interactive 
discussions among research staff and faculty. As our staff attempted 
to create more sophisticated trajectories of our graduates several 
concerns were raised about how we were going to conceptualize and 
measure women's career trajectories. These discussions can be 
emotional because we engage not only a myriad of disciplines and 
Research traditions, but also a variety of personal beliefs. A major 
question that led to many others was: What types of career related 
outcomes are we interested in measuring and how will we measure them? 
Are we to adopt the status attainment model and measure our success as 
educators in terms of the level of socio-economic status or social 
prestige attained by our alumnae? Some staff members have had serious 
reservations regarding the use of socio-economic or prestige indexes 
as valid measures of occupational achievement of women, because of 
disagreement with the assumptions that underlie its construction. 

We all agreed that level of socio-economic status is, by itself, an 
insufficient measure of personal achievement. Indeed, our research 
design is based on the assumption that college outcomes in terms of 
alumnae activities can be expressed in various combinations of alumnae 
achievements. Part of our study involves an attempt to identify 
broader career or life patterns that include in addition to paid 
employment, taking care of family, volunteer work, pursuing further 
education and personal development. From this perspective, a "Mommy 
Track" is a fine track if the alumnae chooses to invest less in the 
job and more in the family, even if it means less opportunity for 
promotion. 

Family: Thin concern about representing other domains of life 
trajectories for women is confirmed by the self-descriptions of the 
alumnae women. Given that 95% of our graduates were employed - 76% 
full- time and 19% part- time, the fact that 75% chose paid employment 
as the primary activity that consumed most of their time is hardly 
surprising. However, it is important to note that women who work 
full-time outside the home, do not necessarily see paid employment as 
their primary activity. Among women who worked full time, 6% 
indicated family was their primary activity, 2%; indicated education 
as a primary activity, and 2%, indicated some other activity as ^ 
primary. Even more striking, is the data for women who have part-time' '* 
paid employment: Only 29% perceived paid-employment as their primary 
activity, and the majority (63%) chose taking care of family as their 
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major activity. We found a strong correlation (r=.52; p / .001) 
between having preschool children and not working full time. This in 
large measure will probably explain why the allocation of time between 
work and other activities, in particular family, tend to change for 
the same women across the interval. Approximately 16% of the alumnae 
did not work continuously for the five years after college, although 
only 1% have never worked. There were also distinct differences in 
these patterns. Five years after Alverno, 17% were working part time, 
but only 8% had continued to work part time throughout the entire 
alumnae interval. An additional 22% combined full- and part-time work 
throughout this period. 

Continuing Education: Continuing education is also an important 
activity area for our alumnae, which is what faculty would expect. As 
many as 51% of the alumnae reported some kind of educational activity. 
This commitment to continued learning includes both formal and 
informal educational activities. Overall, 38% see themselves as 
taking advantage of continued opportunities for continued learning to 
a great or very great extent. Women who have paid employment are more 
likely to perceive themselves as taking advantages of oppoi cunities 
for continued learning, (M ■ 3.32 versus M = 2.82). Turning to formal 
education, five years after graduation, 11% of our participants have 
completed a post graduate degree and 9% are currently enrolled in 
graduate school. We found, as expected a relationship between having 
a higher degree and a supervision position and between the latter and 
salary level. Thus, the occupational and educational trajectories are 
connected. 



Future Plans, Contributions and Unresolved Issues 

Our plans for developing a more sensitive measure of occupational 
mobility include using the participant's own report of further 
education and salary level, as well as her report that promotion was 
a reason for movement from one job to another. We plan to integrate 
all of these sources into categories of career trajectories, through a 
process that will rely heavily upon the coder's expert judgment. We 
are aware, however, that Job titles, full-time/part-time status, 
salary category, highest degree obtained, and other such questionnaire 
variables do not fully enable a precise description of career 
trajectories. 

Also, as we move into our more in depth analyses, we need to examine 
career trajectories for specific professions and major fields in order 
to have a greater opportunity to impact faculty understanding of the 
occupational careers for students in their major so that they improve 
curriculum. 

Finally, a major question is how well we will be able to represent 
family, education, and other activities that elaborate the existing 
paid employment career trajectories. 

To sum up, the questionnaire-based component of the Alverno 
Longitudinal Study of women may provide information that can 
contribute to better understanding different patterns of careers for 
women. We still have to explore appropriate measures of career 
achievement, that are more sensitive to changes in career that are too 
small to be noticed by a national occupation coding scheme, but still 
too important for the individual alumna to be overlooked, if 
successful, this would assist in describing college outcomes for 
women, in a way that could be of interest to external audiences. 
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Nonetheless most of the inmiediately identifiable benefits apply to the 
canpus. For example, the study can facilitate an on-campus 
understanding of the relationship between alumnae perception of 
abilities and college preparation. 

As we look forward to the analysis of alumna abilities, we recognize 
that we move even further into an analysis of the intersection of 
women's public and private life. According to Burbridge (1990), women 
have increased responsibilities in the labor market, but this has not 
been followed by concomitant decreases in family responsibilities. 
What kinds of abilities characterize professionally successful women? 
Do similar abilities emerge in personal life, and if so, what are the 
key abilities that enable women to better deal with their multiple 
public and private responsibilities? 
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Table III-l 



Mean and Standard Deviation of Perceived Frequency That Ability is 
Required on the Job and Strength of Thta Ability Compared to Others 



Ability 


Frequency 


Strength 


Mean 


SD 


Mean 


SD 


o Listen air 9 , respond to the concerns 


H . UJ 




3 80 


74 












o Project self confidence 


4 42 


. 59 


3.57 


.89 


o Think systematically and logically 


4 39 


64 


3.76 


.73 


o Know when and how to get information 


4.34 


.67 


3.72 


.89 


o Demonstrate flexibility when 


4.28 


.70 


3.92 


.77 


unanticipated events arise 




.71 




mm A 

.78 


o Understand the complexity of situations 


4.27 


3.69 


o Take on challenges 


4.20 


.78 


3.90 


.85 


o Communicate effectively in writing 


4. 15 


.85 


3.84 


.94 


o Be open and receptive to different 


4.15 


.73 


3.71 


.89 


points of view 






3.79 


.87 


o Take charge quickly 


4.15 


.91 


o Organize resources and people to 


4. 14 


.92 


3.83 


.78 


accomplish tasks 






3.84 


.79 


o Express/control emotions appropriately 


4. 12 


.80 


o Provide leadership 


4. 12 


.98 


3.77 


.81 


o Express opinions effectively 


4.10 


.66 


3.55 


.80 


o Consider a range of alternative or 


4.08 


.84 


3.72 


.82 


creative approaches 










o Maintain systematic thinking in 


4.06 


.69 


3.73 


.71 


pressure situation 






3.81 


.84 


o Share expertise (e.g., teach, coach) 


4.04 


.92 


o Work cooperatively in groups to 


4.02 


.92 


3.68 


.76 


accomplish goals 










o Initiate action beyond what is called for 


4.01 


.85 


3.89 


.91 


o Address conflict directly and tactfully 


3.91 


.80 


3.49 


.91 


o Implement plans to achieve long-term 


3.82 


.92 


3.45 


.86 


goals 










o Understand and effectively use 


3.63 


1.01 


3.40 


.84 


numerical data 










o Adjust performance based upon feedback 


3.53 


.86 


3.47 


.82 


o Compete when necessary 


3.34 


1.06 


3.29 


.89 


o Make effective oral presentations to grouos 3.33 


1.12 


3.61 


.90 



Note: Frequency 5 -point scale — 1. not at all 5. very frequently 

Strength 5 -point scale — 1. much weaker. ..5. much stronger 
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Table III-2 

Correlation Between Perceived Frequency in Which Ability is Required 
on the Job and Strengt h of Ability Comnared to Others 



Ability. 



o Organize resources and people to 


.56 


accomplish tasks 


.54 


o Initiate action beyond what is called for 


o Compete when necessary 


.54 


o Understand and effectively use 


.52 


numerical, uoua 


.51 


o Take on challenges 


o Take charge quickly 


.50 


o Be open and receptive to different 


.50 


points of view 


.47 


o Understand the complexity of situations 


o Make effective oral presentations to 


.46 


arouDS 


.45 


o Provide leadership 


o Knew when and how to get Information 


.45 


o Communicate effectively in writing 


.45 


o Implement plans to achieve long-term 


.43 


coals 


.42 


o Address conflict directly and tactfully 


o Express opinions effectively 


A 4 

.41 


o Consider a range of alternative or 


A 4 

.41 


creative approaches 


.41 


o Demonstrate flexibility when 


unanticipated events arise 


A f\ 

.40 


o Project self confidence 


o Share expertise (e.g., teach, coach) 


.39 


o Work cooperatively in groups to 


.38 


accomplish goals 




o Maintain systematic thinking in 


.32 


pressure situation 




o Adjust performance based upon feedback 


.32 


o Listen and respond to the concerns 


.29 


of others 




o Express/control emotions appropriately 


.28 


o Think systematically and logically 


.15 



Note: For r _\ .28, p/_ .001; For r=.15, p .05. 
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EXAMPLE IV: ALUMNAE GENERIC ABILITIES 
Synopsis 

Glen Rogers and Mary Talbott describe professional abilities in dynamic 
performance settings. Their work builds upon job competence studies of 
outstanding versus average professionals, some of which have been conducted at 
Alverno (Mentkowski, O'Brien, McEachern, and fowler, 1982; Deback & Mentkowski, 
1982). The Behavioral Event Interview technology that they are reporting on 
may be useful to other college's who are interested in better understanding the 
abilities of their alumni. Working out of a college outcomes framework, they 
are developing a codebook of alumnae generic abilities, and are contributing to 
a taxonomy of abilities required by ill-structured post-college settings. They 
suggest that the dynamic interplay between the research goals of college 
outcomes studies and the existing research and methodology for studying 
abilities will lead to new knowledge. Initial results suggest that effective 
alumnae performance is distinguished by being efficient and having a positive 
influence within a broad organizational context, including the development of 
others. 
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ALUMNAE GENERIC ABILITIES 



Introduction and Background Information 
For Describing the Potential For Research and Practice Contributions 

Our work with the Behavioral Event Interview is focused on the study 
of abilities in ecologically valid settings, where the definitions of 
the problems or tasks being confronted are to a large measure defined 
by the participants. The tasks or problems that the participant is 
confronting are largely ill-structured. Abilities such as 
ego-strength, proactivity, problem finding, and organizational 
strategizing are the topic of study. Numerous methodological issues 
and difficulties must be confronted as we try to take on this task, 
but we have been led to take on this task of studying these kinds of 
abilities in the way that we have by the evaluative goals of our 
college outcomes study. 

We hope that by showing how our research is influenced by our setting 
in a particular college outcomes study, we can make a case for the 
potential of college outcomes studies to contribute to 
conceptualizations of college-level professional and personal 
abilities. In addition to this potential for contribution to research 
on abilities, we expect that other colleges conducting college 
outcomes studies may find the Behavioral Event Interview technology 
that we are using useful for identifying their alumnae outcomes. For 
example, on our own campus we believe that this methodology will 
contribute to faculty understanding of which alumnae abilities 
distinguish effective performance and, thereby, contribute to faculty 
constructions of their curriculum goals. 

We added the Behavioral Event Interview to the battery of measures 
used in the alumnae follow-up of the Alverno Longitudinal Study for 
two reasons. First, we wanted to be able to describe alumnae 
abilities across personal and professional domains with enough 
descriptive richness to impact the Alverno faculty's understanding of 
how alumnae are performing and what abilities they are using. Alverno 
faculty are interested in how well the abilities that students learn 
in the curriculum are manifested in their post-college performance. 
What do the abilities look like in post-college performance settings? 
Do alumnae transform and adapt their abilities? We needed a measure 
of alumnae abilities that had face validity for faculty, that was 
commensurate with faculty's performance-based understanding of 
abilities, and that was descriptively powerful (cf . Mentkowski, 1989; 
Mentkowski & Rogers, 1985; Rogers, 1988). 

Second, we wanted a criterion measure of the effectiveness of alumnae 
performance after college. Such a criterion measure of post-college 
performance is intended to serve as a referent to performance on the 
human potential measures, and the gains on these measures that are 
attributable to the college curriculum. Such a criterion measure of 
post-college performance is intended to span professional domains, to 
include personal and volunteer activities. Alternatives such as 
salary and self -reported ratings seemed limited. How does one anchor 
self -ratings? How does one equate salary across professions, and what 
about volunteer work and family activities? 

The construction of these measurement and description goals and how we 
are trying to carry them out are influenced by Alverno' s educational 
philosophy and principles, as well as being influenced by 
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psychological theory, and the range of methods available for 
identifying abilities (see Figure IV-1). 

For example, Alverno's educational philosophy and principles include 
the educational assumptions that abilities are generic, developable, 
and holistic. We have attempted to construct a generic ability 
codebook in such a way that "the same" ability could be recognized and 
reliably coded in a wide range of naturally varying post-college 
settings, in part because abilities are conceived by faculty as 
transferable across disciplinary domains. 

Because Alverno faculty are concerned with abilities that are 
developable, we have focused on the performance of the ability as the 
unit of analysis, and have not tried to characterize enduring 
personality structures. 

Because Alverno faculty are interested in developing abilities that 
are holistic, they design their assessments to assess abilities 
through performances that are sustained, interactive, open, complex, 
and dynamic. Holistic abilities have a behavioral aspect, a knowledge 
aspect, an affective aspect, a self -perception aspect, as well as a 
motivational or dispositional aspect. Our decision to study 
ecologically valid performances where the tasks are selected and 
defined by the alumnae, and where the tasks tend to be ill-structured 
is influenced by this conception of holistic abilities. Of course, 
the faculty's conception of abilities is already influenced by their 
understanding of what abilities will serve their students after 
Alverno. 

The college includes a liberal arts component in its general education 
curriculum. Liberal arts values, such as sensitivity to individual 
differences, are included in the codebook as code categories. They 
are coded independent of whether or not they appeared to contribute to 
an effective performance. 

Another consideration was the technology that was available to 
identify abilities. Not only did this methodology have to match the 
Alverno faculty's conception of abilities, but it also had to be 
feasible within the frame of the Alverno Longitudinal Study, of which 
it was to be a part. Direct observation of performance in 
pcst-college settings was not feasible, and it was not feasible to 
bring the participants back to campus for performance assessments, 
like an in-basket or set of simulations. The Behavioral Event 
Interview (BEI) provided an alternative — a surrogate to the direct 
observation of performance. The Behavioral Event Interview was 
developed by management and industrial consultants as part of a Job 
Competence Assessment technology (see Figure IV- 1) . 
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Figure tv-1 . Development of a description of alumnae generic 
abilities through behavioral event interviews (BEI's). 
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Job competence assessment is a strategy for identifying abilities that 
distinguish outstanding versus average performers. The Behavioral 
Event Interview (McClelland, 1978) is derived from Flanagan's (1954) 
critical incident technique. The Behavioral Event uses a 
semi-structured interview to develop a detailed narrative account from 
the participant. The participants in this study selected six events 
from their self-defined primary and secondary activity areas where 
they felt that they demonstrated their abilities. The narrative 
account of the participant is probed by the interviewer for specific 
information on what led up to the event, what the participant herself 
did or said, what she was thinking and feeling at the time, who else 
was involved and how they were involved, and what the outcome was for 
the event. 

Job Competence methodology uses the nominations by supervisors or 
peers of outstanding versus average performers within the company to 
identify what abilities distinguish outstanding versus average 
performance. This was not feasible in our study, so, we have focused 
on identifying outstanding versus average performances. We did this 
by using faculty judgments jf the relative effectiveness of 
performances as a criterion for distinguishing which abilities 
contribute to more effective performances. This strategy not only was 
more feasible, but had the advantage of using the standards of one of 
our primary audiences, Alverno College faculty, as an anchor for what 
is considered effective performance for alumnae. Thus, the faculty's 
professional and liberal arts values are infused into the description 
of the effective abilities. 

Although we have relied heavily upon the technology developed through 
job competence studies, we have also found the need to adapt this 
technology to our college outcomes research goals. In particular, the 
expectation that college learned abilities would transfer to a range 
of post-college performances and the need to develop a broad model of 
the development led us to adapt the existing job competence 
methodology to the coding of generic abilities. What has most 
attracted us to the Behavioral Event Interview is its capacity to 
elicit descriptions of ecologically valid and ill-structured tasks. 
This focus on ill-structured tasks (see Kurfiss, 1988) is something 
that we share with the management and industrial consultants using Job 
Competence Assessment technology. It is our belief that 
ill-structured tasks call forth high level abilities. 

Our attempt to develop a generic ability taxonomy and coding strategy 
was also significantly influenced by psychological theories that focus 
on ecologically valid operant performances (see Figure IV-1). For 
exanple motive theories, (see McClelland, 1973; 1987; Winter, 1973; 
1989) have influenced not only our attempt to code achievement, power 
and affiliation motives through the Behavioral Event Interview, but 
also, these motive theories have helped us conceptually organize the 
ability codes under the Entrepreneurial and Interpersonal Ability 
sections of the codebook (see Appendix F) . For example Achievement 
Motive (Efficiency Orientation) is seen as underlying Proactivity and 
Efficiency Actions. Likewise, the Affiliation Motive (Concern With 
Affiliation) is an underlying support to Positive Regard, Sensitivity 
to Individual Differences, Accurate Empathy, and Development of 
Others. 



Researchers studying cognitive abilities through operant methodologies 
have strongly influenced our conceptualization of how to code 
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cognitive abilities. Some of these researchers have studied cognitive 
abilities through operant methodology because they are working in 
college settings like ourselves (Winter, McClelland, & Stewart, 1982), 
but most studied these cognitive abilities through an operant 
methodology because of the capacity of this methodology to identify 
abilities for training and selection in specific work settings 
(Boyatzis , 1982; Klemp, 1988; 1991; Klemp & McClelland, 1986; Sokol 
& Oresick, 1986; Spencer, 1983). George Klemp' s work on defining 
cognitive abilities has been particularly influential, and his work 
can he related to theorist's such as Sternberg (1988) and Gardner 
(1983), who are conceptualizing multi-faceted domains of intelligence. 

Understandably, some of the most influential prior work has been those 
professional BEI studies carried out previously by Alverno College. 
These include a study of nursing professionals (DeBack & Mentkowski, 
1986) and management professionals (Mentkowski, O'Brien, McEachern, & 
Fowler, 1982). The management study (Mentkowski et. al., 1982) found 
that a coding strategy (McBer & Co., 1978) developed from a range of 
job competence management studies could also be applied to outstanding 
managers. The nursing study developed a code book of nursing 
competencies and found that these nursing competencies could 
distinguish baccalaureate from non-baccalaureate nurses. 

In summary, the BEI Generic Ability Codebook that we are developing is 
being derived from a number of sources. Specific codes can be traced 
to a range of Job Competence studies (e.g., Klemp, Huff, & Gentile, 
1980; Klemp & McClelland, 1986; Leles, 1968; Spencer, 1983), to an 
Alverno faculty teacher competence model (Diez, 1990), and to 
descriptions of abilities in Alverno 's ability-based curriculum 
(Alverno College Faculty, 1976, revised 1985). Our goal has been to 
develop a generic ability codebook for the purpose of coding abilitiet 
across professional, as well as personal, activity domains. An 
overview of abilities that are being coded can be seen in Appendix F. 

Cur setting has influenced our research strategy in significant ways. 
It has led us to take on difficult research questions, and to study 
abilities that make a difference in how well alumnae perform. The 
taxonomy of college level abilities, infused by a liberal arts 
conception of the professions and personal life, is arguably a 
different area of study than is defined by existing research (see 
Figure IV- 1) . Job Competence technology is the closest methodology to 
our work, but just as Job Competence technology has been adapted to 
the needs of management and industrial consultant firms, so we expect 
that studies of college outcomes, because of their differences in 
purpose and settings, may lead to innovations in method, and the 
creation of new taxonomies of abilities. 



Brief Overview of the Implementation of the BEI 

Two hundred and sixty five Alverno alumnae were interviewed with the 
BEI . In each case, the participant was asked to identify a primary 
and secondary area of activity. She was then asked to think of times 
she felt effective (two in the primary area and one in the secondary 
area) and times when she did not accomplish what she was intending 
(two in the primary area and one in the secondary area) . The cases 
were then placed in random order and that sequence was followed for 
writing up the interviews. r 
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Approximately 60 cases containing 350 events were written up when the 
faculty validation sample was drawn. The completed narrative accounts 
were initially placed into one of 12 Activity Area categories: 
management, nursing, teaching, family, continuing education, 
developing personal skills and interests, volunteer work, technical 
professions, administrative support professions, customer service and 
related professions, artists, and nursing education. Collapsing 
categories resulted in six Activity Area categories for the faculty 
validation study: management; nursing provider; teacher; taking 
care of family; a range of direct contributors in personal and civic 
activities (developing skills and interests, volunteer work, continued 
education) ; and direct contributors in paid employment (technical 
professions, administrative support, customer service and related 
professions, performing artist) . For the validation study, a second 
random sample of 60 events was drawn from a larger sample that had 
been randomly sequenced prior to being written up. 

In each of the six Activity Areas, two different faculty were asked to 
make overall judgements of effectiveness of alumnae performances 
across ten events within a single Activity Area. Thus Faculty Raters 
were nested within Activity Areas. The ratings of the two faculty 
were combined and then collapsed into three categories so that 
approximately equal numbers of ratings of Most Effective, Effective, 
and Least Effective were created. The 60 events were coded by two 
independent coders blind to the faculty ratings, using a generic 
ability codebook developed for this study, and then consensus coded. 
Faculty combined ratings were crosstabulated with the consensus coding 
of each ability. 



Initial Findings and Contributions 

So far, during the code development stage Category Agreement (see 
Winter, 1973) is .64 when calculated at the level of 32 broad ability 
categories. Development of the coding process for the generic 
abilities involves extensive efforts at defining ability subcodes and 
categories. In contrast, faculty were asked to make judgments of the 
overall effectiveness of the alumnae performance without any 
appropriate description of what might constitute effectiveness. 
Inter-rater correlation between faculty judgments of overall 
effectiveness was clearly positive (r * .39, N « 60). 

Only 31 events were consensus coded at this time for abilities. 
Preliminary cross tabulation of the combined faculty ratings against 
the consensus coding of abilities, yielded 6 of 28 abilities that were 
correlated statistically significantly with faculty ratings. Table 
IV- 1 displays those abilities that appear to distinguish effectiveness 
of performances and Table IV- 2 displays the abilities that do not 
appear to distinguish effectiveness of performance. No claim is made 
for the replicability of these results given the few cases that they 
are based upon. 

As can be seen in Table IV- 1, the performances rated as more effective 
by faculty were distinguished from less effective performances by 
Efficiency Actions, which involve the efficient use of time and 
resources. Another ability that influences effectiveness is 
Diagnostic Information Seeking, which might logically be an initial 
step. Effective performance is also related to an orientation to 
influence that includes the Use of Informational/Expert Influence and 
the Development of Organizational Power. The category of Development 
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of Organizational Power can be coded for either developing 
organizational power through strategic personal contact or by showing 
an understanding of how the organization is functionally structured. 

Faculty judgment of effective performance is not simply related to 
assertiveness and influence of the alumnae. Performance rated as more 
effective are also coded for the Development of Others and Positive 
Regard for others. This is consistent with our expectation that 
liberal arts values may inform faculty judgments of effectiveness. 
More generally, we have tentatively identified what distinguishes 
performances that faculty judge as more effective. 

Taken together, the pattern of abilities that distinguish effective 
performance suggests a performance that makes a difference: it makes 
a difference by being efficient, and having a positive influence 
within the organizational context, including the development of 
others. This might be characterized as a conceptualization of 
influence within the organization, even though the more specific codes 
related to conceptualization and other kinds of analytic thinking were 
coded too infrequently to test this extrapolation. 

Nonetheless, one of the benefits of this Behavioral Event Interview 
Methodology is that it generates highly descriptive demonstrations of 
what the performance of the ability means in context. For example, 
Appendix G shows an example of how Efficiency Actions, Positive Regard 
for others, and Developing Organizational Power are demonstrated in one 
effective performance. The write-up of performance represents the oral 
speech, and therefore does not conform to the conventions for written 
documents . 

These ability codes displayed in Appendix G require some explanation. 
For example, Efficiency Actions was coded for two specific 
subcategories: Lines 11 through 19 show the nurse administrator 
stressing efficiency by balancing task requirements and the needs of 
the nursing personnel. Lines 19 through 45 were coded for identifying 
the constraint to implementing a new and "better" system for managing 
nursing activities, that is, she saw as a constraint on her planning — 
the staff nurses had to have confidence in the new system in order to 
be motivated to enter the data. Positive Regard for others is coded 
(lines 109 through 120) for expressing respect for others who are 
perceived as different. She does this through her action of adjusting 
the talk to meet the needs of different kinds of health care 
professionals. Organizational Power is coded, in this case, for a 
subcategory that codes the alumna's action of seeking an opportunity to 
relate her work to the broader health care system. It should be noted 
that for each of these abilities other subcategory codes exist that 
were not demonstrated 

It is this descriptive power, as reflected in Appendix G, that has led 
the college to use Behavioral Event Interview technology in a number of 
different studies. The descriptive power not only provides examples 
that faculty can use in their teaching, but also conveys to the faculty 
a clear understanding of what post-college performance and performance 
settings look like. This capacity for rich descriptions of performance 
leads us to expect that faculty will be able improve their teaching of 
students based upon the studies identification of key abilities for 
effective alumnae performance. 

SB 
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Abilities Distinc/uishinq Faculty Judgments of Overall Effectiveness 
Bv Frequency That Participant Abilities Were Coded 







Faculty 


Judgment of Effectiveness 


Ability Total Frequency 
Coded Ability Coded 


Least 
(n=ll) 


Effective 
(n=9) 


Most 
(n=ll) 


Efficiency 
Actions 


9 


u 


2 


7 


Development of 
Others 


10 


2 


1 


7 


Use of Informational/ 
Expert Influence 


10 


1 


2 


7 


Positive 
Regard 


8 


1 


1 


6 


Diagnostic 
Information Seeking 


6 


0 


1 


5 


Development of 
Organizational Power 


6 


0 


2 


4 


All Above Abilities 


49 


4 


9 


36 



Note . Table entries are the frequencies that the ability was 
consensus coded by two coders blind to the faculty ratings. This 
table includes data only for those abilities that were statistically 
significantly correlated with faculty ratings of overall 
effectiveness. 
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Abilities Not Distinguishing Faculty judgments of Effectiveness 
Bv Freguencv That Participant Abiliti es Were Coded 



Faculty Judgment of Effectiveness 



Ability Total Frequency Least Effective Most 
Coded Ability Coded (n»ll) (n°9) infill 

Positive Abilities 



Accurate Assessment 


15 


4 


4 


7 


Sees Alternatives 


14 


5 


3 


6 


Concern with Impact 


14 


3 


5 


6 


Initiative 


13 


5 


1 


f 


Pattern Recognition 


11 


2 


4 


c 
0 


Reflective Thinking 


9 


4 


0 


5 


Concern with Affiliation 


8 


4 


2 


2 


Accurate Empathy 


Q 

0 


1 




4 


Self Control/ 


8 


J 


O 

c 


3 


Emotional Stamina 










Use of Socialized/ 


8 


3 


0 


5 


Political Power 








4 


Analytic Thinking' 


7 


3 


0 


Perceptual Objectivity 


7 


1 


2 


4 


Reflectively 


7 


2 


1 


4 


Coordinated Practice 










Sensitive to Individual 


7 


1 


2 


4 


Differences 










Stamina 


5 


1 


1 


3 


Efficiency Orientation 


5 


0 


2 


3 


Specialized Knowledge 


5 


1 


1 


3 


All Above Abilities 


151 


43 


33 


75 





Negative performance codes 






Negative Social 


6 


3 


1 


2 


Emotions 










Lack of Commitment 


5 


3 


0 


2 


to Improvement 


6 








Inaccurate 


3 


1 


2 


Self-Assessment 










All Above Abilities 


17 


9 


2 


6 



Note . The table entries are the frequencies that the ability was consensus 
coded by two coders blind to the faculty ratings* Abilities coded fewer 
than 5 times were omitted. This table includes data only for those 
abilities that were not correlated statistically significantly with 
faculty ratings of effectiveness* 

'This category is coded if hypothetical/causal thinking, 
planning/systematic thinking or conceptualization is coded. 
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Unresolved Issues 

The sample for the study spans across post-college settings and 
disciplines as they vary in the Alverno Longitudinal Study sample, 
rather than being selectively defined by a set of disciplines. This 
wide natural variation required us to create categories and 
subcategories of abilities that were abstract enough to capture the 
essence of an ability and, yet, concrete enough to allow us to 
reliably code functionally equivalent, yet contextual ly diffc .wit, 
manifestations of an ability. Demonstrating that our judgment that, 
although manifested differently, an ability is the "same" presents a 
challenge. Correlating the BEI with the Human Potential Measures 
presents only a partial solution, and relies upon the assumption that 
these measures are an adequate criterion. 

This challenge of trying to code a wide range of abilities across 
domains has led to initial difficulties in getting reliable coding. 
Given the cross domain coding, we are not sure what reliability we can 
get, nor even what would be effective for our purpose. We are still 
in the code development stage, and we expect that the pruning of 
categories will help increase reliability, but this raises the 
question of feasibility of our task. 

We have learned a great deal about how to effectively interview 
(Rogers & Rei setter, 1989) and about how to write-up and code the 
interviews. The interview write-ups have been important for involving 
faculty in the ratings, but they are time-consuming to do rigorously. 
As a result, we have transcribed the interviews, and plan to begin 
coding from the interviews. And yet, we find ourselves again 
confronting a new challenge, coding reliably from the the interview 
transcription. Our goal of coding all of the 1590 alumnae events, 
seems to recede before us. We are constantly in the process of 
re-evaluating what can be accomplished and what sub-goals would be 
meaningful. As is often the case in research using qualitative data, 
the number of cases that can be processed is affected by the depth of 
the analysis and the amount of text to be consumed. 



Contributions 



These preliminary findings support our continuing efforts to develop a 
taxonomy of generic, holistic, and developable abilities commensurate 
with the ill-structured performance settings of college graduates. 
Although we are just beginning, our contribution is related to having 
confronted and taken on this difficult task. In doing so, we are 
identifying issues and method that will enable us to better ask and 
answer the right questions, even if we answer them with other 
questions. We have a demonstrated, with our initial analyses, that 
faculty judgments of effective performance are related to independent 
judgments of generic abilities in a sensible and consistent fashion. 
In other words, the data are consistent with our assumptions and 
expectations. As a result, we have developed more confidence in our 
ability to develop a taxonomy of generic abilities that can be coded 
across professional and personal activity areas. 

A strategy for coding generic abilities that distinguish effective 
alumnae performance could be a significant methodological contribution 
to college outcomes studies. Consider the limitations of the 
alternatives. How does one anchor self -ratings? How does one equate 
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salary across professions, and what about volunteer work and family 
activities? 

We have argued that this college supported work has a potential for 
contributing to a broad taxonomy of post-college abilities, and that 
such a taxonomy may contribute to an understanding of adult abilities 
with implications for educational practice. Any potential for making 
such a contribution, that is external to the college, is dependent upon 
the capacity for this kind of work to also make a difference to college 
itself. In large measure, this means that the work needs to have a 
potential for improving teaching on learning in the institution. Of 
course, this is also true for all of the different components of the 
Alverno Longitudinal Study that are described above. The next section 
addresses some specific connections between evaluation activities and 
curriculum improvement. The section will cover a range of strategies. 
As such, it will include an example of how the BEI management study 
conducted by Alverno had an iirqpact on curriculum development. 
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EXAMPLE V: PROFESSIONAL ABILITIES AND STUDENT OUTCOMES 

Synopsis 

William Richards describes selected sub-strategies used in the 
evaluation of general education outcomes and the major field, with 
attention to the processes by which evaluation, research, teaching and 
curriculum development are integrated in the institution. Results are 
summarized from the studies of professional abilities in nursing, 
management and teaching and their role in each major field's 
evaluation. The current emphasis on describing patterns of change in 
student performance within the major field has helped establish 
procedures for organizing student performance data in relation to 
departmental outcomes and in developing linkages between data 
collection and analysis and curricular elements that can increase the 
quality of overall evaluation and its potential contributions to the 
institution and higher education assessment. 
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PROFESSIONAL ABILITIES AND STUDENT OUTCOMES 
Introduction 

Evaluation has had a central role in the Alverno research program 
since its inception. According to the mission of both the college and 
the office, the process of "validating" Alverno educational frameworks 
creates the institutional need for an evaluative framework in which 
teaching, research, evaluation and curriculum development are 
integrated (refer to Figure 3). 

This dimension of the Office's work — evaluation in general and the 
evaluation of department majors in particular — is shaped by the 
overall educational research program and faculty-designed assessment 
procedures throughout the college. Each of the research and 
evaluation strategies — including those in the Alverno Longititf -1 
Study ~ involve certain applications to curriculum and instru 
The involvement of ORE staff in college functions and committee , and 
in the creation of processes that ensure faculty and research staff 
collaboration, bridges instructional and research units. Rather than 
a passive transfer of information among instructional and research 
units, the Evaluation of the Major Fields is another active strategy 
which implements research and evaluation and concepts of adult 
development, learning and abilities in the study of student 
performance in discipline. (Figure V-l) 

o Evaluation of the major field activities ensure the continuity of 
the research effort and its utility within the college by 
employing some of the same frameworks of data collection and 
analysis (e.g. , the Behavioral Event Interview, self -evaluation, 
the Perry (1970) scheme}. This strategy also benefits from and 
contributes to the faculty' s orientation to working with concepts 
of adult learning and development in relation to curriculum and 
instruction. 

o Using the assessment-as-learning measures inherent in the 
curriculum and the Office's own analytic strategies, the 
evaluation of the major field is currently focusing on preparing 
aggregate and developmental portraits of student performance in 
the context of a discipline, examining inter- and 
intra- individual patterns of change. This frame of analysis 
differs from the departments' on-going evaluations and the 
self-studies which might be required in curriculum review or 
accreditation. Here we take another kind of "look." It is based 
on systematic data collection of aggregated student performances, 
supported by analyses from ORE, as a another 
perspectives from which to examine the curriculum. 
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Figure V-l. Evaluation of general education and major field outcomes. 
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o The approaches allow for testing and validation of curricular 
elements in terms of both (a) selected models of adult 
learning and development and (b) discipline-based 
frameworks. 

o Because this is an internal evaluation process, integrated into 
the college's mission, it benefits from certain elements, but 
also must meet certain standards (cf., House, 1986, Sonnichsen, 
1987) . As an internal activity, it is possible for the 
e valuators to understand conditions with greater depth and to 
have an investment in the use of findings. At Alverno, 
evaluation is integrated into the academic aspects of the 
operation rather than the administrative. Furthermore, it 
involves faculty members and departments as active contributors 
and maintains their investment in the results and the process. 
This relationship requires that external frameworks be 
incorporated in the data collection, including those that involve 
undergraduate education, the theories and methods of adult 
development, learning and abilities, and the principles and 
practices of hi$ ler education research and evaluation. Thus, 
part of the challenge which defines this type of evaluation 
activity is the need to resolve quality and utility — that is, 
objectivity, rigor, and professional standards with usefulness 
and feasibility. 

The strategy for evaluation of general education and the major fields 
can be broken open into sub-strategies using data from several 
different sources: 

General Education- Alverno Mission 

Eight abilities 
Domains of knowledge 

Major Fields- Description of the Major 

o Learned societies 

o Professions 

o Other college faculty 
Literature review 
Interdisciplinary dialogue 
Multicultural frameworks 
Competence chair review 
Credentialing groups 
Professional performance 

and alumnae studies 



Three of these will be discussed here: general education outcomes : 
pmfftRs^nrwl wi-m^w aimed at refining discipline based outcomes; and 
patterns of student performance in relation to discipline-based 
outcomes. (Figure V-2) 

The remainder of this section provides a brief overview of these 
studies, some applications from the current work on student outcome 
patterns, and a review of the challenges and unresolved issues which 
emerge for the evaluation of the major field. 
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Figure V-2. Evaluation Strategies 
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Internal an d External Contributions: The Studies 

The Evaluation of Outcomes in G eneral Education. The Alvemo 
Longitudinal Study has focused on evaluation of the general education 
curriculum and has included connections to discipline-based research 
frameworks. For example , in an earlier section, Reisetter has 
discussed ego, moral and cognitive development using measures from 
these research traditions and discussed the possibilities of making 
contributions to these discipline-based theories. But these 
contributions to disc ipline-based theory are only feasible within 
collage outcome studies If they also make a contribution to curriculum 
development and teaching within the colleg e. Although the evaluation 
of general education outcomes differs from the evaluation of 
individual major fields, this aspect of the research work is integral 
to more specific evaluations and sets a foundation from which 
subsequent evaluations build. 

In some cases, the methods used and even developed in the study 
continue to be used across campus. Human Potential Measures have 
been and will continue to be used in future analyses. For 
example, the Learning Styles Inventory (Kolb, 1984) is regularly 
administered (Mentkowski & Giencke-Hall, 1982). Developmental 
theory — for example, Perry's (1970) scheme of intellectual and 
ethical development, Kohlberg's (19Clb) Rest's (1979a, 1986) 
schemes of moral development — continues to inform curriculum 
development and practice, even though the thinking of the faculty 
does not rely on any one theory. There is a dynamic interplay 
between the work of our office and the work of other departments. 
For example, in some cases, methods such as the Behavioral Event 
Interview were being developed simultaneously for use in the 
research efforts and in other departments. As a result of this 
dynamic interplay, the office is prepared to adapt the 
instruments and analytic techniques to current use in curriculum, 
and there is a readiness among faculty to use the language, 
concepts and findings which are produced. 

This approach also produced a baseline of data on achievement and 
the persistence of general education outcomes. For example, as 
Deemer describes earlier in this document, the Perspectives 
Interviews provide descriptions of longitudinal changes, the 
developing use of criteria, and the ability to self assess; the 
data from the Perspectives Interviews and other measures provide 
foundations for examining student progress within the major field 
as well as differences in individual and aggregate progress. 

Furthermore, the ability to define the extent of general 
education outcomes becomes critical in the evaluation of major 
fields so that relationships among general education outcomes , 
ma-tor field outcomes and instruction in maior course work can be 
explicated. 
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Professio nal mh^Hm. As we turn to more specific evaluations of 
individual major fields, we find that the research efforts of the 
Longitudinal Study continue to provide frameworks and relevant data. 
Using methodology derived from the Behavioral Event Interview, and 
cooperative research involving faculty and research staff, the 
Professional Studies examined relationships between professional 
abilities and curricular programs within the college. The studies 
resulted in models of professional abilities for use in other 
educational contexts while contributing internally to curriculum, 
instruction and assessment procedures. 

Between 1979 and 1983, the studies were conducted with three divisions 
with professional preparation programs: Education, Nursing, and 
Business and Management. The general purpose for the second two 
studies was to study the on-the-job performance of outstanding 
professionals who were npt our graduates. In education, data were 
generated from literature review, interdisciplinary team work, and a 
student performance review. From these data, three competence models 
of interrelated abilities were developed. Each division evaluated the 
central elements in the department's curriculum against a model. This 
activity focused primarily on the major outcomes which were identified 
in each department and around which the curriculum was organized. 
These models were also disseminated through the publications cited 
below. 

o Teacher Education! A Developmental Model of Teacher 
Competence 

The work involved four years of study with faculty members in the 
college, across disciplines, and reviews of student performance, 
as well as a review of the teacher education literature. It 
resulted in the articulation of five competences as developable 
abilities which were begun and fostered through course work and 
refined over the professional career (Diez, 1990) . These 
abilities were prepared as "maps" (Figure V-3) which described 
the status of each as: 

- the basic/beginning abilities of educators entering the 
field (e.g., student teaching, first year) 

- developing abilities characteristic of teachers with 
several years experience 

- advanced abilities which mark a master teacher who has 
demonstrated professional depth and development. 

The "basic/beginning abilities" were designated as the broad 
target for teacher preparation and were eventually designated as 
the advanced outcomes for the the Teacher Education program. The 
basic/beginning information from the maps was incorporated with 
data from other sources in the development of various instruments 
for rating fieldwork performance for students. Specifically, 
basic/beginning abilities are the source for the criteria in the 
comprehensive assessment used to assess performance in the 
student teaching semester. 
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Figure V-3, Model of the development of teacher abilities (1984), 

Ability 1: CONCEPTUALIZATION. Integrating content knowledge with educational framework and a broadly-based understanding of the liberal arts in order to 
plan and implement instruction. 



BASIC/BEGINNING ABILITIES 

Applying analytical skill to the integration of 
knowledge base and psychological/philosophical 
foundations of education 

• Shoving command of subject areas 

• Presenting subject matter in conceptual context 

Content knowledge base e.g., math, 

language arts 
Linking Educator frameworks e.g., cognition, 
frameworks development psychology, curriculum 

theory Liberal art skills e.g., reflection, 

epistemology inquiry approaches 
Developing sensitivity to learners as individuals within 
the group as a whole 

= Making links between developmental theory and 
concrete individuals, e.g M using appropriate depth of 
subject matter, showing understanding of student 
behavior and motivation 

• Planning material both to meet learners current 
needs and to lead to the next level of development, 
e.g., using logic in the development of subject 
matter, relating subject matter to previous work, 
helping students to relate subject matter to "real 
life 0 experiences 

• Analyzing effect of class activities on both 
individuals and class as a whole 

Developing an understanding of the system within 
which one works as educator 

• Recognizing communication networks in the 
organization 

• Learning the history and philosophy of the 
organization 

• Becoming aware of developmental sequences in the 
system 



DEVELOPING ABILITIES 

Building increased ability to use the knowledge base 
as a resource in facilitating learning 
Continuing to expand and develop own knowledge 
base in all these areas 

Increasing sensitivity to learners in t group and as 
individuals 

• Trying to meet learners' perceived needs 

• Trying to stretch student to the next steps in their 
development 

• Sensing multiple possibilities, i.e., the array of 
things that might happen, that one might do 

• Predicting the impacts of various strategies 

• Evaluating plans in relationship to outcomes in a 
systematic and ongoing way 

• Constantly relating information to frameworks 
Refining understanding of the system 

• Knowing what others have done/are doing (e.g., in 
previous classes, in concurrent classes) 

• Knowing the whole educational sequence and 
learners in relationship to it 

Developing tolerance for ambiguity by making links 
between the ideal and the real 

• Constantly responding to unpredictable source of 
input 

0 Recognizing that one cannot demand resolution 



ADVANCED ABILITIES 

Showing commitment to a style of thinking within 
one's discipline 

• Taking responsibility for clear presentations 
providing for common experiences 

• Finding ways to make links with students' ongoing 
experience, letting go of one's involvement with 
content in order to promote class dialogue 

Show high ability to pick the right strategy for the 
right situation 

• Calling forth higher level learning 

• Facilitating high level discussion (e.g. beyond the 
"internalize my ideas and frameworks" sort) 

• Figuring out the patterns in interaction to help the 
learner bring analytic thinking to a new level 

Modeling an adult learning process 

• Showing own growth and reflecting on own growth 

• Shaping the relationships between the concrete skill 
processes focused on and "own lives" to show 
educational environment as transforming 

• Acting with creative awareness of how to use the 
system to support and promote learning skills 

• Operating with autonomy, able to singlehandedly 
manage multiple schemes, individuals and 
interactions 
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O Developing a professional Competence Model for Nursing 

Using the procedure of Job Competence Assessment , Nursing faculty 
and Office of Office of Research and Evaluation staff conducted 
Behavioral Event Interviews with nurses in a three health care 
settings to establish the professional competence model. Results 
showed that competences apply significantly more often to nurses 
with education at the baccalaureate level or higher and to nurses 
with more experience (more than 5 years) . Findings helped 
establish the validity of the core elements in the Nursing 
curriculum. (See Figure V-4, also DeBack & Mentkowski, 1986) 
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Figure V-4 . Nurse performance by education and experience. 



GENERIC COMPETENCE MODEL FOR EFFECTIVE NURSING PERFORMANCE: A CODEBOOK' 
I. CONCEPTUALIZES M. EGO STRENGTH 



Conceptualizes is coded positively when the nurse forms a 
concept by recognizing the relationship between two 
different pieces of information in the following ways: 

A. The nurse explains a rationale for her thoughts or 
action by identifying the two pieces of information 
and the concept used to explain the relationship It is 
also coded positively if the nurse indicates her 
understanding of the relationship even though she 
does not explicitly name the concept formed. 

B. The nurse uses a concept to recognize that, while all 
the individual data are within normal limits, the 
pattern of data indicates that something is wrong. (It 
is not coded if the pattern is so routine that anyone 
would be expected to have noticed that problem.) 

C. The nurse brings to bear non-routine resources to a 
problem or applies routine resources in a creative 
manner. 

Conceptualizes is coded negatively when: 

0. The nurse presents two pieces of information from 
which a concept could be expected to be drawn but 
tails to draw the relationship or organizing principle. It 
is also coded negatively if the nurse presents the 
theoretical knowledge which could be applied but 
does not apply it (Using this code overrides any 
Helping or Coaching behavior resulting from the 
negative conceptualization, even though these would 
otherwise be coded.) 9 

E. The nurse demonstrates a pre-occupation with a 
specific task to the exclusion of the higher organizing 
principled). (Using this code overrides any Helping 
or Coaching behavior resulting from the negative 
conceptualization, even though these would . 
otherwise be coded,)* 

II. EMOTIONAL STAMINA 

Emotional Stamina is coded positively when the nurse 
performs her responsibilities despite strong emotional 
reaction to a situation. The nurse must mention that she 
had a strong emotional reaction and there must be 
evidence that this did not interfere with her performance; 
i.e., she must show evidence of overcoming a strong 
emotional response. Emotional Stamina Is coded positively 
when: 

A. The nurse simply does not allow emotions to interfere 
with performance by controlling anger, overcoming 
fear, or responding calmly when attacked. 



Ego Strength is coded positively when the nurse shows 
evidence of being able to withstand confrontation, 
disagreement, or disapproval to persevere in her judgment 
or is able to use assertiveness despite disagreement or 
disapproval. (An element of risk must be involved for the 
nurse in order for Ego Strength to be coded: confrontation 
or disapproval alone is not sufficient for coding.) Ego 
Strength i t coded positively when: 

A. The nurse fulfills her responsibility at the risk of 
incurring the disapproval of another (supervisor, 
patient, peer). 

B. The nurse admits a weakness, mistake, or lack of 
knowledge while recognizing the importance of 
remedying it. 

Ego Strength is coded negatively when the nurse shows 
evidence of abandoning her responsibility when meeting 
disagreement or disapproval. Ego Strength is coded 
negatively when: 

C. The nurse abandons a responsibility when she 
perceives a barrier, taking no steps to test the reality 
of the barrier.' 

0. The nurse changes her behavior in the face of 
disapproval by another, or acts to avoid disapproval 
rather than to fulfill responsibilities/ 

E. The nurse acts out of a need for the approval of 
others.* 

F. The nurse feels ineffective and helpless, unable to act 
as a result.* 



IV. POSITIVE EXPECTATIONS 

Positive Expectations is coded positively when the nurse 
expresses the belief that another person, or people in 
general, have basic worth or ability to perform. Positive 
Expectations is coded positively when: 

A. The nurse expresses the belief that people are worth 
teaching. 

B. The nurse sees others as generally competent. 

C. The nurse reserves judgment until all evidence is in 
regarding policy violation. 



Emotional Stamina is coded negatively when: 



Positive Expectations is coded negatively when: 



27B From: DeBack & Mentkowski (1986) Does the VOL 25, NO. 7, SEPTEMBER 1986 

baccalaureate make a difference?; Differentiating 
nurse performance by education and experience > 
q Journal of Nursing Education , 25, p. 278-80. 
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EigiiCfi g=4 (continued! « Nurse performance by education and experience 



B. The nurse presents evidence that an emotional 
response interfered with her performance." 

C. The nurse gives evidence that bottling up a strong 
emotional response interfered with her performance." 



V. INDEPENDENCE 

Independence is coded positively when the nurse takes an 
action when there is no external pressure to do so. 
Independence is coded positively when: 

A. The nurse takes an advocacy role tor a patient or 
subordinate. 

B. The nurse takes responsibility tor her own judgment 
and acts independently. 

Independence is coded negatively when: 

C. The nurse avoids taking responsibility for her own 
judgment and'or gives up." 



VI, REFLECTIVE THINKING 

Reflective Thinking Is coded when the nurse identifies and 
reflects upon her own behavior, feelings, or beliefs and 
their consequences. It may indude reflecting upon a 
weakness or mistake, and must result in the nurse 
showing new insight or searching for new insight. 



D. The nurse looks down on a person as being 
incapable." 

E. The nurse treats a person as a member of a class 
rather than as an individual (stereotypes) or 
generalizes from individual to group behavior." 

VIII. INFLUENCING 

Influencing is coded when the nurse shows a concern for 
changing the attitude or behavior of others tor a purpose. 
(There must be evidence that these others are not seeking 
the same goal the nurse is seeking for them.) 

Influencing is coded when: 

A. The nurse attempts to persuade someone to follow 
her example. 

B. The nurse provides a rationale tor the desired 
behavior, including appealing to a higher motive. 

C. The nurse persuades by a variety of strategies or 
searches for one strategy from alternatives. 

D. The nurse uses a strategy to refocus from negative 
emotions to more constructive issues. 

E. The nurse provides valid information in order to 
change attitude or behavior. 

IX. COACHING 

Coaching is coded when the nurse uses any of a variety of 
strategies to instruct, train, or encourage patients or 
subordinates to accept more responsibility for themselves 
or for their jobs. Coaching can also be seen as a 
specialized form of INFLUENCING in thai the strategies 
used are ways to influence combined with the motive to 
Increase the other* responsibility. 

Coaching is coded when: 

A. The nurse gradually increases the responsibility tor 
tasks or for self-care. 

B. The nurse rewards desirable behavior or gives 
positive feedback in other ways. 

C. The nurse provides information to increase the 
other* responsibility. 

D The nurse fits a task to a perceived interest in a 
subordinate or patient. 



VII. HELPING 

Helping is coded when the nurse takes action to help a 
patient or subordinate personally or demonstrates a 
concern for the other person* needs. (There must be 
evidence that both the nurse and the person she is helping 
are seeking the same goal.) 

JOURNAL OF NURSING EDUCATION 
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Figure V-4 ( continued 1 . Nurse performance by education and experience 



Helping is coded when: 

A. The nurse listens actively and attentively. (There 
need not be evidence that the person she is listening 
to recognizes this behavior on the part of the nurse.) 

B. The nurse searches tor methods of establishing 
rapport. 

C. The nurse provides valid information. 

D. The nurse acknowledges the needs of another. (This 
requires evidence from the person whose needs are 
being acknowledged that she tie recognizes the 
acknowledgment.) 

E. The nurse searches to understand the other's 
perspective. 

F. The nurse acknowledges the needs ol another. (This 
does not require evidence Irom the person whose 
needs are being acknowledged that she he 
recognizes the acknowledgment.) 

'Tebtt 5 is drawn from a reseerch repon by Mentkowski. DeBeck. Bishop. Allen, and Bianton. (1980) "Developing a Profession! Competence Model tor 
Nursing Education* which was funded by a grant from the National institute of Education. The report is available from Alverno Publications. 3401 South 
39th Street. Milwaukee. Wisconsin 53275. or ERIC Document Reproduction Service (No. ED 239 565). 

Note: An asterisk after a behavioral descriptor in competencies U II UK IV. and V is a reminder that a behavioral descriptor stated for positive categories is 
not used tor negative coding. 
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O Developing a Professional Compatance Model for Business 
Education 

As with the Nursing Division, business and management 
faculty and Office of Research and Evaluation researchers used 
the Job Competence Assessment methodology to ascertain the 
abilities which distinguished outstanding women managers and 
executives (Mentkowski, O'Brien, McEachern & Fowler, 1982; also 
discussed in Mentkowski, 1988a) . The study had three major 
outcomes: 

o A competence model of effective managerial performances that 
can serve to improve management education programs 
(Figure V-5) 

o A pool of over 500 behavioral examples set within particular 
contexts that can serve as instructional tools, assessment 
criteria and feedback for management students 

o Better advice for women students seeking examples of 

careering and professional development and how it relates to 
effective performance in the managerial role. 

After reviewing the competence, the business and management 
faculty incorporated several findings. For example, expanded the 
outcomes all students are expected to demonstrate to include one 
which dealt with taking initiative in identifying and solving 
problems or taking a leadership role in helping the organization 
take advantage of growth opportunities. They also developed a 
number cf instructional and assessment techniques to address 
critical incidents from on-the-job performance. 
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Fi gure V-5 . Hypothetical aodel of competence in voaen managers and executives. 
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From: Mentkowski (1988). Paths to 

integrity: Educating for personal 
growth and professional performance. 
In Srivastva and Assoc., Executive 
integrity . San Francisco: 
Jossev-Basn. 
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Student Performance and Maior Field Outcomes . in the Professional 
Studies, professional performance expectations provided a lens 
through which to examine and contribute to major field outcomes. In 
the current evaluation emphasis , different ways of describing student 
performance are being developed as an additional perspective and a 
means for contributing to the development of curriculum and 
instruction. 

Several institutionalized processes are already in place to provide 
departments with evaluative information (assessment-as-learning for 
individualized development, credentialing and program evaluation; 
department review, self-study for accreditation, etc . ) . In the 
current approach, systematic questions are being raised based on the 
patterns of student performances. This involves a stepping back from 
the immediate operation of instruction and day-to-day student 
performance assessment to examine the aggregate, longitudinal patterns 
of student performance: 

1. To what extent can student performances on faculty-designed 
assessments in the major be aggregated and used to portray 
student progress through the curriculum? The accumulation and 
organization of these data provide a basi? for examining 
particular student performances in relation to the major outcomes 
as well as a basis for charting intra- and inter-individual 
patterns of change. 

2. Will an integrated picture of student performances yield the 
advanced outcomes in the discipline? What implications can be 
found, in this type of analysis of student performance for the 
major field unit (i.e., curriculum, department, division)? 

3. What broad differences can be described among students in the 
patterns which characterized their progress through the major? 
What implications do these patterns have for curriculum and 
instruction as the departments attempt to meet individual, needs 
learning in the discipline? 

While these questions are of particular consequence tc the individual 
institution, they may also have more general importance 

To what extent can such findings contribute to higher standards 
of post secondary instruction? 

And what methods can be developed to increase the usefulness of 
institutional assessment? 

This dimension of evaluation provides a common ground where curriculum 
and instruction, institutional frameworks, and student development and 
learning are integrated and therefore a place v.here the perspectives 
of faculty members, researchers and administrators meet. It is a 
context in which significant questions can be raised and addressed. 
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The work we report on here began in Fall 1990 with the Natural 
Sciences and the Education Divisions. The first step was to construct 
the portraits of student performance which would ultimately provide 
the basis for collecting data and structuring an analyses. This 
required considerable collaboration between and among department 
faculty and Office of Research and Evaluation staff in order to make 
the constructions adequate and usable. These, then, provided maps from 
which student performance data were identified. (See Figure V-6 for an 
example. ) 

Working with the faculty, different questions were posed by the 
different majors. 

o Education Division 

In Education, the concern was with students who were having 
uncommon difficulty in meeting performance expectations, 
particularly as they entered student teaching. Using samples of 
students who were clearly excelling and those who were 
experiencing uncommon difficulties, trajectories of student 
performance were prepared to examine different patterns. The 
actual measures included narrative assessments on performance 
(including a Behavioral Event Interview) , quantitative ratings 
from f ieldwork placements and the portfolio assessment procedures 
which are used in making decisions about student teaching. The 
data emphasized the role which personal maturation can play in 
performance and serves as a basis for continuing discussions on 
how curricular and instructional elements can respond to 
individual differences in maturity and inconsistencies in 
performance. 

In itself, the portfolio assessment presented a unique 
opportunity for students themselves to define and present their 
individual trajectorj.es in the major. The students' portfolios 
were assessed by practitioners in a half-day event which included 
group and individual interactions for students and assessors, 
lengthy discussions and consultations. The results demonstrated 
the ability of such an activity to put students' work in a 
longitudinal perspective, to clarify differences among individual 
students, and the values inherent in and implications of their 
diversity. In addition, the need to prepare the assessors for the 
kinds of roles which they would play as well as to further refine 
these roles was emphasized. 



EXTERNAL 
ASSESSMENT 
SEQUENCE IN 
THE 

UNDERGRADUATE 
TEACHER 
EDUCATION 
MAJOR 



TEACHER CANDIDATE PROFILES- 
BEHAVIORAL EVENT INTERVIEW 
ASSESSOR REVIEW 



> Commitment to 
Teaching 

> Conceptualization 

> Diagnosis 

> Coordination 



> Communication 

> Interaction 

> Range of Experience 

> Strengths 

> Areas for improvement 



INTEGRATED COMPETENCE SEMINAR 
ASSESSOR REVIEW 

> Taking a Position 

> CommunlcationandAudienceAwareness 

> Considering Alternatives 

> ThWdngThrough, Organizing... 

> Group Problem Solving 

> Defining Problems and Planning 

> Appropriate Actions 

> Aesthetic Response 



FIELD EXPERIENCES -OJf-Campus Assessments Social Interaction LEVELS 4, S 



DEPARTMENT SUPERVISOR 

> Behavioral Observations 

> Evidence of planning 

> Commitment to attaining goals 

> Use of social interation skills 

> Communication skills 

> Integration of theory and practice 

> Classroom/behavioral management 

> Attention to ind differences of learners 



CO-OP TEACHER 

> Behavioral Observations 

> Professional conduct 

> Attention to Learners 1 self-image 

> Use of social Interation skins 

> Communication skills 

> Conceptual/analytic skills 

> Effective problem solving 



COURSES: L EARNING EXPERIENCES 



FRESHMAN YEAR 



SOPHMORE YEAR 



figure V-6. 
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PORTFOLIO ASSESSMENT 
ASSESSOR REVIEW 

> Materials -presentation, foundation in 
content and theory, logical 
development 

> Personal presentation -communication, 
* motivation, respect for others 

>Strengths & targets for development 

>ReadtoessforStudenlTeachlng 

SELF EVALUATION 

> Goals for Student Teaching 



DISCIPLINE OUTCOMES 
EhmtntMryEducMtton 

> CONCEPTUALIZATION: Integrate content 
knowledge wkh educational frameworks and 
broadbasedunderstanding of liberal artsln 
plm^arxiimplement^ 

> DIAGNOSIS: Relate observed behavior to 
relevant frameworks In order todetermlne and 
Implemertleamingprescriptk^ 

> COORDINATION: Manage resources 
effectively to suppon leaning goals 

> COMMUNICATIONS: Uses verbal, nonverbal 
andmecfiamodeatoevtabish the environment 
of the classroom and to structure and reinforce 
learning 

> INTEGRATIVE INTERACTION: Act with 
professional values as a situational decision- 
maker, adapting to the changing needs In the 
environment in order to develop studetns as 
learners 



SELF-EVALUATION 

> Achievement of Objectives 

> Social Interaction-Class Management Skills 

> Instructional Behaviors and Student Response 

> Modifications from Original Plans 

> Performance Strengths 

> Performance Weaknesses 

> Means to Improve Effectiveness 

> Working In a multicultural context 



STUDENT TEACHING 
ASSESSOR REVIEW 
>Conceptualzatk>n 
> Diagnosis 
>Coorcfinatk>n 
>Communteatlon 
> Integrative Interaction 



AND ASSESSMENTS 



JUNIOR YEAR 



SENIOR YEAR 
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O Natural Sciences Division 

For Natural Sciences, the study focused on a senior 
self-evaluation in which students were asked to reflect on the 
full experiences of undergraduate work and the role which 
Alverno's curriculum played in their growth. In a pilot effort, 
the self evaluation essays for three students were rated using a 
coding system originally developed in regard to William Perry's 
(1970) scheme of intellectual and ethical development of college 
students and employed in the Longitudinal Study (Mentkowski, 
Moeser & Strait, 1983) . The results provided an interesting 
opportunity for examining the interaction of general and 
discipline-based outcomes, relations between assessments and 
targeted outcomes, and increasing the use of evaluation measures 
within the curriculum. 

Language and concepts appear in the text of some students' essays 
which suggest that the student values a set ideas even if she has 
not yet internalized them. This leads to questions about 
relationships between college and departmental outcomes, as well 
as how or when these need to be clarified in evaluation. The 
persistence of the fundamental, general education outcomes is of 
critical concern. But the impact of the discipline-based 
curriculum may appear superficial if students are focusing on the 
most fundamental themes which unite their undergraduate years in 
such essays. 

In this small sample of essays, students ranged considerably in 
their approaches. For example, one student offered a fairly 
surface-level description of the general curriculum impact: 

My ability to analyze and to problem solve have also 
developed to a high level of skill. . . 

At Alverno I have achieved the ability to demonstrate 
all these different skills that I have just mentioned. . . 

I will strive to be an effective part of my community, 
to use my knowledge to help others and to better the 
world around me. 

While another student demonstrated her more in-depth assimilation 
of the disciplinary content and her integration of social roles 
and academic expertise: 

As a junior, I was focused on finding who I was and what 
I wanted to do with my life. This not only involved 
contemplations about my major and career, but I also 
became concerned about my position in society as a Black 
woman. . . {An internship) not only broadened my 
perspective on the environmental concerns of businesses, 
but also gave me a chance to achieve one more thing I'd 
once doubted I could do. . .and that was be a 
professional. 
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I'm concerned with what I learned in my biology and 
business courses more than ever. . . {but} I remind myself 
that being able to recite my education class by class is 
not as valuable as understanding the relationships 
connected to each class or being able to apply my 
knowledge in a useful situation. 

The first student describes an appreciation of her multiple 
abilities, the other describes herself impelled toward personal 
and professional commitments. 



To some extent, helping students make more thoughtful reflections 
can be addressed in the protocols for the self assessments/self 
evaluations used by faculty in the major to elicit student 
integration of their abilities, before entering the major and/or 
afterward. This kind of instrument development draws on the 
procedures used college-wide to develop and validate assessment 
instruments. However, the recognized potential for using these 
student performances as part of program evaluation may sharpen 
attention to certain aspects of the assessment protocol (e.g. , 
how to elicit meaningful feedback and reflection on learning 
experiences) . 

As discussed in the introduction to this section, when faculty 
members read through the essays and the analyses, they clearly 
recognized common frameworks among the college's principles/ 
practices and the analyses. This sets, in part, a unique context 
for moving between the results and the curriculum in an 
evaluation mode. 

In themselves, these studies are not yet complete evaluations. At 
this stage, they are efforts which (a) establish procedures for 
organizing and analyzing student performance data in relation to 
departmental outcomes; (b) demonstrate the feasibility of these types 
of studies (including joint faculty-researcher efforts at dealing with 
student performance in her major field) , and (c) develop linkages 
between data collection and analysi3 strategies and curriculum 
elements that can increase the quality of overall evaluation activity 
and the potential for impact. For a faculty already doing on the 
spot, on-going, and year end evaluations of student achievements and 
for an internal evaluation office, these are critical next steps. 
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SUMMARY AND OQMCLUSICMS - UNRESOLVED ISSUES 

In this section, the perspective has been taken that higher education 
assessment, if it is to have a significant impact, must integrate 
concepts of adult learning and development with issues of curriculum 
and instruction. Evaluations of major fields become a particularly 
fruitful means for bringing together different perspectives, giving 
faculty members and researchers an equal investment in student 
learning and development. This cnn result in risky, difficult 
confrontations, but working with the hard questions can lead us into 
more productive territor* 

What may be unresolved issues from a research point of view may appear 
to be resolved for faculty members in the course of normal operations, 
or, at least, to pose different kinds of questions. For example, if a 
student is not meeting performance expectations because of her lack of 
personal maturity, faculty must still create learning experiences 
which will help the student as well as challenge their own ability to 
stimulate her development. But, more to the point, the types of 
concerns or paradoxes which characterize research may need to be dealt 
with differently from an evaluation perspective specifically because 
the work of curriculum proceeds regardless of these resolutions. 
Instruction may be hampered by the lack of resolutions or their 
inadequacy, but it proceeds nonetheless. Therefore, in the following 
paragraphs, three concerns are identified which may be resolved in 
practice at particular institutions but which remain continuing issues 
in the larger discussion around "good" institutional assessment in 
higher education. 



The Internal Evaluator 

Understanding distinctions between internal and external evaluations 
is useful, but the key role of internal evaluation in institutional 
assessment activities ma)ces it more important to illuminate the 
benefits and challenges which emerge from an internal operation in a 
particular institution. A better understanding of how the tension 
discussed earlier between quality and utility can be resolved should 
contribute to the standards under which the most productive work will 
occur. At least one element in this is a collateral relationship 
between research/evaluation and curriculum/instruction, with a mutual 
investment in improving student learning and development. Earlier in 
this document, Rogers and Talbott have discussed hew successful 
faculty involvement increases the potential for usefulness just as 
faculty leadership to the value of the professional abilities models. 
However, the fuller the integration of evaluation into the institution 
the more important it may become to evaluate the contribution of 
evaluation to the institution. The extent to which external frames of 
reference or criteria are used from educational specialities (such as 
educational research or evaluation) in various analyses is a partial 
answer, but the issue remains. 



Separating ~ and Dealing v th ~ the Outcomes of the College, the 
Discipline and Personal Maturity 

Faculty seek to increase the learning of students despite a range of 
individual differences in maturity, and the college, the discipline 
and personal maturity that occurs during college all have distinct but 
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combined impact on student outcomes. By defining these separate but 
combined effects and their interactions, an evaluation seeks to bring 
the most useful information into curriculum development. This seems 
implicit in higher education evaluation, but that doesn't mean that it 
can be easily achieved to serve the needs of faculty who are concerned 
about educating their students in the major field. In addition, as 
noted in this section in the discussion of the student self 
evaluations, some of the outcomes of undergraduate education may be so 
fundamental that they should have priority within more discipline 
specific outcomes. To some extent, this may also be a methodological 
concern, but it is one which will benefit from more attention. 

The Evaluation Role in the Context of an Assessment Movement 

Given its current place in history, a number of questions can be 
raised about how the current attention to higher education assessment 
differs from or relates to evaluation in higher education. An 
oversimplification of the concepts (e.g., evaluation deals with 
decision-making, assessment with measurement) may not be that helpful. 
We have presented a range of examples from research and evaluation 
activities here in order to ask the question in more depth, and to 
pinpoint the corresponding contributions of general research and 
evaluation practice to work in institutional assessment. At a time 
when higher education assessment is still developing conceptual 
frameworks, a great deal will be gained from a committed reflection on 
a range of approaches and methods. 
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SYMPOSIUM QUESTION FOUR: HOW CAN COLLEGE OUTCOMES STUDIES 
SIMULTANEOUSLY CONTRIBUTE TO AN INSTITUTION'S PURPOSES AND TO THE MORE 
GENERAL PURPOSES OF EDUCATIONAL RESEARCH AND FOSTSECONDARY PRACTICE? 

Alverno Question Four is the same: How Can Alverno College Outcomes 
Studies Simultaneously Contribute To An Institution's Purposes and to 
the More General Purposes of Educational Research and Postsecondary 
Practice? 

Synopsis 

Marcia Mentkowski provides a capstone description of lessons learned 
and unresolved issues that address simultaneous contribution to an 
institution's purposes and to the more general purposes of educational 
research and postsecondary practice. 

Multiplicity of measures and theoretical frameworks are essential for 
a complex picture of college outcomes because of the limits of any 
single strategy or measure. Longitudinal results are useful for 
faculty understanding of student patterns long-term; performance-based 
measures used in evaluation of general education and the major field 
yield more usable data for program changes. Questionnaires are useful 
in demonstrating the degree to which alumnae perceive that personal 
and professional goals are met, but fail to address the complex 
performance outcomes achieved through studies of alumna and other 
professional abilities. 

With a range of approaches and strategies and a corresponding set of 
diverse findings, faculty are better able to interpret growth patterns 
across domains in relation to curriculum. They are better able to 
understand how learning is constructed by students, which abilities 
are performed by alumnae, and how ability models of outstanding 
professionals can link to evaluation of student outcomes in the major 
field. External audiences are also likely to find contributions to a 
general picture of human potential useful as a backdrop to their own 
curricular development efforts, if they share a mission for teaching 
and student development. 

There is a dynamic interplay between interactive, interdisciplinary, 
integrated research and evaluation approaches. Applying the 
combination of strategies needed by faculty to improve curriculum is 
essential. This, together with the intent to meet simultaneous 
internal and external contributions, can create paradoxes among 
conflicting criteria. Insights from dealing with paradox can 
illuminate institutional assessment as an emerging field, and identify 
criteria for quality assessment. 

Contributions to Educational Research Methods and Discipline-Based 
Theory and Methods: Lessons Learned and Unresolved Issues 

The five examples of contributions to a general picture of 
development, abilities and learning illustrate the dynamic interplay 
of research and evaluation approaches. There are a range of lessons 
learned and unresolved issues: only one of each is listed here for 
each example. Thoso related to symposium question four regarding 
simultaneous contribution are also discussed. (Refer to Symposium 
Questions 2 and 3 in Figure la, "Lessons Learned" and "Unresolved 
Issues" . ) 

ERIC 1 1 5 
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Example I: Cognitive, Moral and Ego Development Trajectories 

Lesson Learned: Developmental trajectories differ across 
domains and timeframes. 

Unresolved Issue: Who changes and why? What are best methods 
for analyzing intra- and inter-individual change patterns? 

College outcomes include complex, holistic human abilities that 
develop differentially during college and afterward. Curriculum has 
more of an impact on the development of some of these abilities than 
others during college. But clearly, both older and younger students 
are achieving benefits, irrespective of the degree to which we can tie 
these benefits to the curriculum. More interesting are the patterns 
in these abilities in relation to each other. Now that we are 
analyzing a fourth data point — after college — our appetite is 
whetted for more information. The different trajectories of 
cognitive, moral and ego development presented in this symposium are a 
case in point. 

Equally important is the time frame for development. There are 
differences in when these abilities develop during the college years, 
and now we have a partial window on how they develop afterward. The 
recognition measure of critical thinking reported on in this symposium 
can satisfy some who question whether students in an ability-based 
curriculum show upward growth on a traditional measure. That such 
change seems unrelated to curriculum is puzzling when some other 
production measures of critical thinking show such changes. 

Moral judgment develops during but not after college; ego development 
occurs after college but not during. Why this happens this way for 
this group of individuals is not clear, but other researchers provide 
some clues. Rest would argue that once an individual leaves a formal 
educational setting, reasoning about moral issues would tend to 
plateau. We confirmed this. Loevinger suggests we would not see 
changes after college. We did not confirm this prediction. In sum, 
developmental trajectories show that cognitive development increases 
during and after college; moral judgment increases during college and 
plateaus afterwards; and ego development shows no change during 
college, but growth afterwards. 

It is clear that definition and measurement of college outcomes needs 
to include a range of dimensions: cognitive/intellectual process, 
affective/socio-emotional process, perceptions, motivation, and 
performance. Most educators are struck by the difficulty of any 
attempt to separate these aspects. Yet attention to each dimension in 
turn may be necessary to enable students to integrate them later on. 
The form integration takes across the ten-year period under study is 
of interest to us, and one of our next analysis questions. 

Unresolved is the question of how to best illuminate patterns of 
change in each of these dimensions. Once we untangle inter- and 
intra- individual differences, will it make sense to look at these 
dimensions once again in the aggregate? 
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EXAMPLE II: Self-Sustained Learning and Development 

Lesson Learned: Knowledge and performance are linked in 
student constructions and attributed to curricular elements; 
understanding criteria leads to self-sustained learning. 

Unresolved Issue: What student perspectives and curricular 
elements relate to gains in cognitive , moral and ego 
development? 

It seems likely that our best understanding of integration of 
abilities, learning and development may be derived only from the 
student '8 own perspective. To our surprise, our studies of student 
perspectives — known too often in the literature as "only 
self -report " — had the greatest benefit for our faculty colleagues. 
Insight into how the student thinks about learning, abilities, and her 
own development — and particularly to what elements of the curriculum 
she attributes her learning, were helpful to faculty on a variety of 
fronts, and also helped create a learning inventory that is currently 
helpful to students and faculty alike. 

So far, descriptions of learning outcomes from the perspectives 
interviews mirror to some extent similar descriptions by David Kolb 
and William Perry; the linking between knowledge and performance in 
student perspectives is a new finding. A perspectives interview data 
base is useful in providing a picture of these outcomes in relation to 
the curricular elements that, in the students' mind, contribute to 
independent learning, making relationships between abilities and their 
use, and using different ways of learning. Development in students' 
understanding of criteria — the behavioral descriptors that faculty 
and students use to evaluate performance — lead to self-sustained 
learning. 

Coming up with feasible and rigorous methods is a continuing 
challenge, and we look forward to completing our analyses of student 
and alumna perspectives on learning and developmental domains, using 
better qualitative analytic technologies. 

In the future, we expect to analyze the quantitative trajectories of 
cognitive, moral and ego development in relation to the more indepth 
picture of individual development that the interviews provide. Here 
is where our most likely contribution to developmental theory will 
occur. Here we will be able to pinpoint elements in the curriculum 
linked to cognitive-developmental patterns, and search for learning 
strategies linked to development during college and afterward. 
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Example III: Career Trajectories of Women 

Lesson Learned: Older and younger women achieve management and 
professional positions after college. 

Unresolved Issue: What alumnae measures consider the 
intersection of personal and professional, public and private 
contributions? 

Our graduates have much to teach us about how women fare after 
college. The questionnaire data is positive; Alvemo women are 
satisfied with their majors. They report that education prepared their 
well for life after college, particularly for life-long learning, and 
they give the college even higher marks for liberal arts outcomes than 
for career-related benefits. 

Older and younger women achieve management and professional positions 
after college. Thus, our results can contribute to the picture of 
women's experience in a managerial roles that, compared to nursing and 
teaching, are non-traditional for women. What are the abilities 
younger women use to manage career/family conflicts, especially if 
they have young children (Mentkowski, 1983)? Do some of the older 
women experience a glass ceiling or age discrimination as they try to 
move out of clerical positions after college? To what degree are 
older women stymied by secure financial benefits that a 
" job-in- the-hand" provides? 

The study of career trajectories has been hampered by a lack of 
indicators that describe mobility for women adequately. Clearly, 
analysis of the interplay between personal and professional, private 
and public responsibilities is a center stage issue for liberal arts 
colleges preparing women for professional careers. 
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Example IV: Alumnae Generic Abilities 

Lesson Learned: Some generic abilities distinguish effective 
alumnae performance. 

Unresolved issue: How well do generic ability codes cross a 
wide range of personal and professional activities? 

Descriptions of alumnae generic abilities are more interesting to us 
at the moment than questionnaire data. But the difficulty of creating 
generic codes that cross personal and professional contexts is 
daunting at times. 

A picture of performance from the alumnae ability, studies tells a more 
complex story, one that rests on events alumnae relate about times 
they were successful, and times when they did not meet their 
objectives. Faculty ratings of these events pinpoint those where 
alumnae performance did not meet faculty expectations, as well as 
situations where they did. Clearly, pictures of performance have the 
potential to revise faculty expectations, and also to set standards 
for performance. As more studies like this one are conducted, higher 
education may be in a better position to describe how alumnae actually 
perform, and refine or revise claims for college outcomes accordingly. 

EXAMPLE V: Professional Abilities and Student Outcomes 

Lesson Learned: Curriculum development is informed by a 
dynamic relationship among coherent, diverse research and 
evaluation strategies. 

Unresolved Issue: How does personal development interact with 
disciplinary outcomes? What form does personal and 
professional integration take? 

Studies of professional abilities built three models for each of three 
majors: nursing, management and education. These have been useful in 
informing curriculum, and have formed a basis for a more generic set 
of abilities that we are now using to study alumnae abilities , Faculty 
use these alumnae and professional ability models to evaluate 
disciplinary outcomes in the major field. Here faculty compare 
patterns in performance from a series of faculty-designed, capstone 
assessments across time, to these externally-derived performance 
m odels . 

Pictures of human potential from cognitive, moral and ego development 
measures provide another backdrop. Clearly, curriculum development is 
informed by a dynamic relationship among coherent, diverse research 
and evaluation strategies. Those that use data from faculty-designed 
performance assessments are most useful for improving curriculum. 
Unresolved, and of ongoing interest in how personal maturity interacts 
with disciplinary outcomes, and what form integration of disciplinary 
and other abilities take. 
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Broad Contributions to General Educational Research Methods 

and Postsecondary Practice 

Lesson Learned: Multiple frameworks and methods yield a 
useful, complex picture of college outcomes. 

Unresolved Issue: How achieve long- and short-term benefits 
simultaneously? 

In the previous sections we intended to demonstrate that pictures of 
human potential informed by college outcomes studies of development, 
learning and abilities can contribute to a more general understanding 
of college student and alumna gains. We know more than we did about 
developmental trajectories, and visiting educators seem as fascinated 
as we are with these patterns. The ability models have been widely 
requested by faculty from other institutions, partly because they were 
developed from outstanding professionals who were not limited to our 
graduates. To what extent pictures of abilities of Alverno alumnae 
will be of interest to external audiences is unclear; however, 
educators building descriptions of abilities — from critical thinking 
to self -assessment, may find these descriptions useful because they 
are derived from performance in personal and professional, public and 
private domains. 

Clearly, no single college outcomes approach or strategy yields the 
complex picture of human potential that can contribute to educational 
practice. While the complexity can be overwhelming at times, we have 
learned that generating both long- and short-term benefits for 
curriculum development are important; it is best if both kinds are 
occurring t imultaneously. Yet this has been difficult to achieve. 

Our longitudinal analysis of change strategies were helpful for 
immediate curriculum evaluation early on when we were collecting data 
on current students, and less helpful for that purpose as we conduct 
studies of alumnae. Yet now we are realizing the developmental 
trajectories that are of interest to our faculty. Now we have 
patterns over the 10 year period a faculty member feels responsible 
for: a student's college years and how she turns out about five years 
later. 

These broad patterns are no substitute for studies of current students 
in the major field that illuminate individual trajectories in the 
development of disciplinary abilities, and give important information 
on how to refine curriculum. Yet, we have found that current student 
outcomes may be difficult to interpret without a backdrop of 
information about alumnae abilities five years out, and abilities of 
outstanding professionals who are not our graduates. Faculty 
expectations of students are the product of many sources of 
information. Our research and evaluation office provides a broad 
picture against which faculty can review their day-to-day observations 
from student performance on faculty-designed performance assessments. 

Lesson Learned: An institution's educational frameworks shape 
a conceptual base for assessment. 

Unresolved Issue: How will different conceptual bases for 
assessment impact college outcomes studies? 



ERIC 



UNDERSTANDING ABILITIES, LEARNING AND DEVELOPMENT 1991 AERA SYMPOSIUM 



Simultaneous Internal and external benefits accrue from coherent, 
diverse multiple approaches and strategies. Educational frameworks 
inform question-asking; discipline-based theories inform 
interpretation of findings. An educational institution's 
practice-based philosophy and principles form a conceptual base for 
assessment. These are, in turn, informed by external theories and 
methods as findings from multiple studies converge. College outcomes 
studies, conducted for purposes of institutional assessment/ are 
likely to reflect differences in institutional purpose, just as they 
are bounded by setting and sample characteristics. We can look 
forward to a great diversity in findings, and an enriched picture of 
human potential, where student outcomes are related to institutional 
purposes and practices. 

Lesson Learned: Contributions to methods can occur when 
educational purposes and research methods are inseparable. 

Unresolved Issue: What institutional processes link purpose and 
method? How does method change as a result? 

Lesson Learned: Contextual ly valid findings contribute to 
general theory and practice when they link practice-based 
frameworks with discipline-based theory in a dynamic setting. 

Unresolved issue: What are principles of institutional 
assessment that is interactive, inter-disciplinary, 
integrative, and intends simultaneous internal anl external 
contributions? 

In a setting that employs educational framework-driven institutional 
assessment, purpose and method are inseparable. Methods bend to 
purpose. This can result in changes in method or suggest alternative 
ones. We begin to question the philosophy and values underlying 
methods as a result. For example, how do we define "validity" in such 
a setting, where a dynamic interplay between research and practice is 
a goal? Who asks the questions becomes as important as how inquiry 
proceeds. Who can use the information is as essential as the 
soundness of the interpretation. The findings that result, tested in 
a variety of ways, contribute because they link practice-based 
frameworks with discipline-based theory in the kind of dynamic setting 
that is characteristic wherever teaching and learning is a major goal. 

We have learned to think of our research and evaluation activities as 
reflecting "applied" and "action" research approaches that are 
problem-driven and practice-based. However, the dynamic character of 
our work in this setting suggests other characteristics: 
"interactive," "interdisciplinary," and "integrative." A combination 
of these descriptors defines institutional assessment for us. But we 
struggle to cnsate the processes that engage all parties of interest 
and that will achieve this ideal. A coherent set of principles of 
educational framework-driven institutional assessment as we practice 
it is emerging from this experience. The effort can be overwhelming 
at times , as we confront the diversity of perspective inherent in a 
liberal arts faculty and built into our research and evaluation team. 

Lesson Learned: Interdisciplinary research teams apply 
criteria from various theories and methods, for various 
purposes. This creates paradoxes. 
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Unresolved Issue: Which criteria apply? How will these shape 
institutional assessment? 

Working with multiple purposes, designs, instruments and methods may 
seem like the researcher's dream — generating results that can be 
triangulated on each other and validated more immediately through 
practice. Ihis diversity can often skew us off into a lack of focus, 
and draw attention away from the primary purposes for the research. 
Balance is provided by interactive processes that enable our research 
and evaluation team to confront the immediate and long-range goals of 
a faculty every step of the way. 

This diversity generates conflict often as does the diversity built 
into the team via graduate training from several institutions, from 
several disciplines. Gone are the days when one could bury one's self 
in an independent research program, work alone, and come out 
periodically for peer critique or the weekly graduate seminar. 
Instead, one works with other team members and faculty daily on 
projects within and outside one's immediate expertise. 

Researchers new to our team quickly *et go of lingering assumptions of 
this liberal arts college as a quiet place where a common purpose 
results in "group-think. " Rocky discussions result from diverse 
theoretical and method frameworks. Conversations can erupt as the 
philosophical and value oases of different perspectives move against 
each other. A deep and sustained regard for diverse perspectives, and 
for each other, contribute to constructive interaction. 

We have found that the conflict such collaboration generates often 
benefits external audiences. Individuals from different disciplines 
bring to bear the questions from "outside" that challenge the deeply 
held beliefs of another. These conflicts cause stalemates or get 
over personalized at times. The immediate needs of a faculty provide 
an incentive to quickly get to the heart of an issue, decide how the 
conflict will change approaches and interpretations, separate the 
wheat from the chaff, and move on to the next set of issues. 

Conflict is constructive when the team can make progress toward 
meeting its goals and can decipher which criteria are being applied 
during critique and argument. There are many dilemmas or challenges 
that we face if we are to meet the criteria for studies that meet 
internal and external contributions simultaneously. 

For example, utilization aid generalizability criteria are often at 
odds in meeting both internal and external purposes. This is not the 
only paradox. Meeting the criterion of coherence among educational 
frameworks and purposes, and research and evaluation approaches and 
strategies, and at the same time building in diversity of theoretical 
frameworks, methods and measures generates paradox. Nevertheless, our 
educational frameworks, which flow from liberal arts principles and 
multiplicity of perspective within a coherent community of learning, 
remains an ideal. So we continue to work with coherent, diverse 
approaches and strategies for research and evaluation. 
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Which Criteria Apply to College Outcomes Studies Conducted for the 
Purposes of Institutional Assessment? 

Implied in these lessons learned and unresolved issues is a set of 
goals and standards. What should we expect of ourselves? Should an 
institution meet standards of institutional scholarship computable to 
the standards it sets for individual faculty scholarship? Are the 
standards similar? What are the responsibilities of institutional 
scholarship? Where and how is an institution expected to contribute 
to the field of higher education? Only as an example or case study? 
Merely as a source of practice-based theory and method? Or can an 
institution be considered a contributor to general theory and method? 

What scientific criteria should be applied to educational research, 
evaluation and measurement methods that are used in this new kind of 
application? What is the potential for contributing results to 
discipline-based theory and method building? 

So then, which criteria apply (refer to last column in Figure la)? 
Existing criteria from educational research, evaluation, measurement, 
and institutional research are sources. Because higher education 
assessment has created a new context for applying criteria, it may be 
premature to assume which criteria apply, and how and in what kinds of 
settings "good" assessment will accrue as the result. The exercise of 
examining which criteria apply by looking at current college outcomes 
studies in a variety of institutions can be helpful. We do not want 
impossible standards to overwhelm initial efforts to design 
assessment, or to propose criteria too far in advance of our 
collective experience. But we feel that this ongoing exercise is 
important to the ultimate goals of institutional assessment as an 
emerging field (Mentkowski, 1989). 

We highlight two combinations of criteria for higher education 
assessment as those that have provided the most challenge. This is 
not because any one criterion, on its own, is intractable. Rather, 
it is this particular combination in college outcomes studies that 
creates challenges. Paradoxes are highlighted when an institution's 
mission calls for both internal and external contributions. 

We invest in defining criteria that sometimes are in conflict, because 
the criteria we and others use for critique determine whether that 
critique can be constructive. For example: 

a) are results usable and generalizable? 

We have just presented examples which deal with usefulness and 
application of findings for both internal and external purposes; 
sometimes utilization and general izability criteria conflict 
(Mentkowski, 1989) . Let us look at a second combination, that of 
coherence and diversity (Mentkowski, 1990). 

b) are designs, purposes and methods coherent, and do they also 
provide for diverse and multiple frameworks and 
perspectives? 
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The apparent paradox is between the coherence ana diversity that is 
inherent in a rethinking around the purposes, functions and outcomes 
of undergraduate education. This reconceptualization seems to be 
initiated both inside (Study Group on the Conditions of Excellence in 
American Higher Education, 1984) and outside higher education by 
society's primary beneficiaries (Ewell, 1990). 

These developments include efforts toward (Mentkowski, 1990): 

o connecting curricular components in the face of a growing 
diversity of courses and majors; 

o integrating educators' experience with their disciplinary 
expertise; and 

o promoting interdisciplinary approaches in assessment design. 

Implications for institutional assessment include developing 
approaches that reflect concerns for both coherence and diversity. 
Some coherence in curriculum — via sequenced courses or a more 
coherent design like Alvemo's — calls for symbiotic educational 
frameworks and assessment methods. This is balanced by diversity of 
perspective inherent in the liberal arts and in a multidisciplinary 
faculty and staff who employ various theoretical frameworks. 

Degree of curricular coherence is an important factor when the goal is 
to evaluate curriculum effects. Are there, at best, identifiable 
curricular principles in the minds of the students; is there at least, 
a rationale for learning? 

Students may experience the curriculum quite differently. To what 
degree are perceptions related to actual involvement? Can we relate 
outcomes to the curriculum? We share Astin's (1991) view that one 
test of a curriculum means demonstrating causal relationships between 
curriculum and student learning outcomes. 

This concern for degree of curricular coherence is paralleled by equal 
concerns for graduating students whose abilities are demonstrated in 
unique, creative ways. Curriculum has another goal, to maximize 
individual differences. What kinds of designs will enable distinctive 
student outcomes to emerge, with the richness of intra- individual 
patterns intact? 

Two other combinations of criteria provide additional challenges, and 
are raised implicitly in the examples presented in this paper. These 
are: 

c) is measurement valid and reliable? 

d) are methods feasible and rigorous? 

On the last point, Terenzini (1989a) argues that "whether one is 
dealing with design, measurement or analytical issues, it will be well 
to remember that campus-based assessment programs are intended to 
gather information for instructional, programmatic and institutional 
improvement, not for journal publication" (p. 661). Our own view is 
that this is traditionally the oase, but that an institution with a 
mission, for example, to develop human potential would be expected to 
contribute to a more general picture of human potential, and that the 
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Information collected as part of the educational research component of 
its institutional scholarship efforts would be publishable. 

Terenzini goes on to say that "methodological standards for research 
publishable in scholarly and professional journals can probably be 
relaxed in the interests of institutional utility and advancement" 
(1989a, p. 662). What doss that mean? That institutional scholarship 
in the service of improving learning can somehow apply "relaxed" 
standards or criteria? He believe that institutional scholarship and 
assessment can serve discipline-based theory and method. We would 
take the stance here that it is hardly a matter of relaxing standards 
or being less rigorous. Often it is a matter of realizing what level 
of analysis one is dealing with, and how systematic the information 
needs to be to answer a question. We find that one has to meet more 
kinds of criteria that are often conflicting, and that the difficulty 
is deciding which criteria apply. 

In sum, we meet, choose, combine, and adapt criteria from several 
educational specialties (e.g., educational research, evaluation, 
measurement) . This means applying criteria that do not work against 
the characteristics of institutional assessment in this setting — 
where findings are expected to have "contextual validity" (Mentkowski 
& Rogers, 1985; Rogers, 1988). This often means working with methods 
and instruments that are less well established in the literature, 
adapting, for particular purposes, instruments and methods for 
particular purposes which are new combinations of purpose and method, 
and in some cases, creating new methods. 

We expect our symposium discussants to speak to this question: 

o To what extent is educational research that is internally 
driven, by institutionally defined purposes and questions, 
valid and useful to the educational research community, as a 
way of addressing larger questions about teaching and 
learning? 

Joan Stark and Jon Wergin will discuss strengths and limitations of 
the approaches just presented, and identify issues that these and 
other college outcomes researchers will need to resolve in order to 
benefit higher education practice more widely. They will also make 
observations on the potential for contributions to educational 
research from college outcomes studies, and suggest ways that AERA's 
Division J can provide both professional critique and opportunities to 
evolve criteria for higher education assessment. 
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Appendix A. Degcription of the Current Population . 
Characteristics of the Student Body 

Alverno's enrollment — 2,414 — has tripled since the beginning of 
the Longitudinal Study. As a women's college with a program of 
liberal arts and professional preparation, its enrollment is 
characterized by a wide range of ages, with many women who are 
returning to school and also first generation college attenders, and 
recently a a wider ethnic diversity. In the last decade, the 
college's population of weekday students more than doubled, with 
predictable impact on the characteristics of the student body: 

Approximately 24% of Alverno's students are under age 23, 
compared with 20% in 1981, while the proportion of those 
over 30 has remained at about 53%. Weekend College students 
are significantly older, with over 71% over the age of 30. 

Minority enrollments have tripled in this period, with their 
representation increasing from 9% in 1981 to about 19% in 
1990. 

Of the students reporting a religious preference, 56% are 
Catholic, 19% Lutheran and 15% other. 

In Weekday College, 60% of the women are single-never 
married; 20% are married. In the Weekend College, 30% are 
single-never married and 43% married. 

As the enrollment has grown, the number of students who 
reside on campus has also, although not as much as the 
number of commuters. Resident students have made up 9% of 
the enrollment for several years. Forty-seven percent of the 
current students are full-time; the proportion rises to 66% 
for the Weekday College and drops to 30% for the Weekend 
College. In the Weekday College, over 50% of the majors are 
in three program areas: Education (29%), Business and 
Management (13%), and Nursing (13%); 27% of the Weekday 
students are in the departments of Art (9%), Professional 
Communications (9%), and Psychology (9%). The Weekend 
College program is based on three majors: Business 
Management (63%), Professional Communications (30%) and 
Nursing (7%). 

Retention and Graduation Rates. Based on a study of enrollment and 
graduation over the last 10 years, the college estimates that: 

o Over 80% of the students in the first semester of their first 
year will return for the second semester {NB: Marlene Neises is 
sending me the report on this but this is the way she described 
the results to me) 

o Approximately 66% of the students graduate within 5-6 years of 
entry. 
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Appendix A (continued) . Degflrtption of the Currant Population. 
flonpariaon with National Statistics 

Alvexno's enrollment shows both similarities and differences in 
comparison with the national statistics on post secondary students 
(National Center for Education Statistics, Digest of Education 
Statistics, 1989). 

The age distribution among Alverno's students is 
considerably older, although national enrollments among 
older students have grown. In total, students over 30 years 
of age rose from 21% of the enrollment to 25.7% between 1975 
and 1985. However, among women students, those over 30 
represented 22.6% of the total in 1975, but 30.0% in 1985 
(compared with over 50% oi Alverno's current students) . 

At the national level, the ethnic diversity of student 
enrollments has changed considerably in the last decade. 
For the total of students attending public institutions, 
minority representation increased from 17.9% in 1976 to 
21.2% in 1986. This trend is somewhat lower for the total 
enrollment in private institutions (15.4% in 1976 to 19.7% 
in 1986). In this regard, Alverno's population (19%) is 
more similar to the national total than might be expected 
from regional characteristics (e.g. , minority enrollment 
across Wisconsin is 6.5% of the post-secondary total). 

In 1986, at the national level, Business was the most 
frequently reported major for undergraduates, representing 
17% of the total and 16% of all women undergraduates (13% of 
Alverno's current enrollment). Business and Management 
represented 24% of all bachelor degrees awarded (22% of 
those awarded to women) ; Education represented 9% of total 
(13% for women); Nursing equaled 3% of total (6% for women). 

Recognizing that methods for computing graduation rates 
vary, the most comparable statistic, in terms of Alverno's 
documentation, comes from the Department of Education High 
School and Beyond survey. Based on the responses of a 
sample of 1980 high school seniors who entered four-year 
college programs, 46% of those in public institutions and 
55% of those in private institutions had received their 
degrees by 1986. While Alverno's current graduation rate is 
somewhat higher (66%), it includes some transfer students 
with prior education. 
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Appendix B. Description of the Longitudinal Sample. 

The domain for the longitudinal sample was composed of all women 
entering the college in the years 1976 and 1978. As discussed below, 
the rules for eligibility and the rates of participation result in 
certain changes in the sample across the assessment periods. 
Approximately 350 women were stable members of the sample, with 
demographic characteristics distributed as follows: 

- 40% were traditional age students — that is, entering 
college at 17-19 years of age; 30% were over 30 years of age 
at entry. 

- They were 94% white and 6% black. 

- Among those reporting a religious background, 63% were 
Catholic . 

- 31% were married at the time of entry; 58% were single, 
having never married, and 12% were divorced, separated or 
widowed. 

- 25% came from families where the parents had less than a 
high school education; 12% had mothers who had earned 
college degrees and 16% had fathers with college degrees. 

- 35% reported that their mothers had no employment or careers 
outside the home and only 15% reported that their mothers 
had managerial or professional positions compared with 27% 
whose fathers held managerial or professional positions. 

In terms of educational background: 

- 24% had high school grade point averages lower than 2.5 and, 
for 16%, they were higher than 3.5. 

- Nearly half (49%) entered with no prior college, while 32% 
transferred with over one semester (i.e., more than 15 
credits) . 

During their time at Alverno: 

- Over 75% were full time students. 

- Only about 17% lived on campus. 

- 43% majored in Nursing, 31% in Business Management, 7% in 
Education, 3% in Communications. 

- Of the 351 students who entered at Time 1 without prior 
college, 146 — 42% — graduated from Alverno within six 
years. Of the total entering students, 46% graduated within 
six years. 
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Appendix B (continued) . Degoription of the Longitudinal Sample. 
The sample is divided into three cohorts , as described above: 

- 25% from the Weekday students entering in 1976. 

- 32% from the Weekday students entering in 1977. 

- 43% from the Weekend students entering in 1977. 
Distinctions among these cohorts involve age and prior education: 

- Weekend College students tended to be older; they were more 
likely to have been married and have children. 

- Weekend College students and older students in all cohorts 
were more likely to have had some prior college but also to 
have had lower high school GPA's. 



Rfipresentflhlveness of the Lon gitudinal Sample 

Because additional data were collected from the participants in the 
Longitudinal Study, some comparisons are not possible between the 
sample and the current enrollment. However, in terms of general 
demographics, there are several differences: 

The age distribution in the sample was somewhat younger: 30% 
had entered after the age of 30 while over 50% of the 
current enrollment is over age 30. 

There were fewer minorities — 6% compared to 19% today. 

They were slightly more Catholic — 63% compared to 56% 
today. 

Approximately 30% of the total students in both the sample 
and the current enrollment attended college while married 
and the trends among the Weekday /Weekend and 
Full-time/Part-time studonts are similar for both groups. 

The distribution of selected majors for the sample differs 
considerably, with a much heavier emphasis on Nursing and 
much less on Education. The sample's distribution reflects 
the prevailing choice of majors at the college at the time 
these students entered. For example, in 1977, Nursing 
represented 53% of the Weekday College majors and 28% of the 
Weekend; Business and Management represented 4% of the 
Weekday and 48% of the Weekend; Education was 15% of the 
Weekday and Professional Communications was 25% of the 
Weekend. 

Full time and resident students were more frequent in the 
sample — 75% and 17%, respectively — than in the current 
enrollment — 47% and 9%, respectively. These figures also 
seem to reflect general trends in the college's attendance 
patterns. 
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Appendix B (continued) . Description of the Long^ivUnal ga^ia 

In addition, a substantially larger percentage are currently staying 
in the program and graduating within a six-year period. 

These distinctions become important considerations as the data are 
analyzed and inferences made regarding the current population. 



Eligibility and Participation 

The rules of eligibility were designed to increase the study' s 
capacity to monitor individual changes in relation to learning 
experiences at Alverno. 

1. The participants were required to be on campus during the first 
three assessment periods. 

2. They had to have participated at the Time 1 assessment. 

3. For the Time 4 assessment — five years after college — they had 
to have participated in at least two of the first three 
assessments (i.e., 1 and 2, 1 and 3, or 1, 2 and 3). 

Because of the first rule, the eligible sample dropped considerably 
between the Time 1 and Time 2 assessments. At the Time 1 assessment, 
conducted during the campus orientation for new students, 706 students 
were eligible from the 1976 and 1977 cohorts combined. In the two 
years between the assessment periods, 313 students left campus: 

142 left the program in the first semester for their 
respective cohorts (NB: In particular, 1976 was an anomalous 
year for the college, and most of this group of leaders was 
from the 1976 cohort) 

27 graduated in the two years between the first two 
assessment periods 

144 transferred or left the program after attending for at 
least one semester 

Between the second and third assessment periods (i.e., about two years 
for both cohorts) , an additional 66 women graduated and 68 transferred 
or otherwise left campus. 

Therefore, while sample attrition was large between Times 1 and 2, the 
eligible sample remained at about 300 for the remaining three 
assessment periods. 

Participation ranged from 97.0% to 85.8% at each of the times of 
assessment, averaging 90.2% ovjrall. 
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Appendix C: Developing Perspectives on the Role of Criteria for 

Student Understanding of Independent 
Learning and Self -Assessment . 



o 



WHAT VALUE AND BENEFIT DO ASSESSMENT CRITERIA HAVE FOR STUDENTS? 
1 CRITERIA MAKE INDEPENDENT LEARNING POSSIBLE] j fCRITERIA MAKE SELF-ASSESSMENT POSSIBLE I 



... from content to abilities 

... from vague to explicit to flexible 

interpretation 
... from external to internal self-assessment 



{ . • . from grades to criteria 
{ ... from quantity to quality 
j . from opinion to evidence 



• Sees learning objectives as vague directions for 

what to learn 

• Finds explicit directions too picky 

• Sees learning objectives as directions for how 

much content to learn 



BEGINNING STUDENT 



• Sees competences or abilities as directions for 
what to do 

9 Asks for explicit directions for what to do to 
perform, to get validated, or to "pass" 
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• Sees assessor judgments as arbitrary and vague and 

dependent on factors beyond own and assessor's control 

• Finds explicit assessment criteria too picky 

• Sees assessor judgments as based on standards for how 

much to learn 

• Sees number or letter grades as the standards for how 

close you are to learning enough of the right answers 



• Sees criteria as feedback on strengths and weaknesses 

but as vague with little meaning for "passing" 

• Sees that assessor judgments are based on criteria, 

but finds interpretation of criteria arbitrary and 
vague and dependent on personal opinion of the 
assessor and self 

• Often doesn't understand why validated or not 

• Sees criteria expressed as percent of correct response 

• Worries about motivation to achieve where can pass by 

just getting by 
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Appendix C (continued): Developing Perspectives on the Role of 
Criteria for Student Understanding of Independent 

Learning and sei f -Assessment t 



DEVELOPING STUDENT 



Sees that criteria given ahead of time tell you 

what to learn and what to do 
Asks for explicit learning objectives and 

criteria 

Sees abilities as steps in a process that you 

use in school and personal life 
Sees learning as a process (you learn how to 

learn and it doesn't disappear afterwards) 



Sees that feedback on strengths and weaknesses provides 
explicit information on progress and success 

Sees criteria as a framework for feedback and 
self-assessment 

Asks for explicit criteria 

Motivated to achieve by explicit criteria 

Rejects grades as a source of information on progress 
and success 



• Sees criteria as providing a picture of the 
ability to perform 



i 



• Sees criteria for assessment as more flexible and 
ambiguous, as more open to interpretation 



"ADVANCED STUDENT" 



• Sees criteria as one part of a process for learning and assessment 

• Sees abilities as frameworks for performing and criteria as a picture 

of the ability for performing and for self-assessment 

• Sees criteria as a cognitive framework for learning, that enable 

transfer of learning 

• Sees criteria as being met in more ways than one, and uses in a 

flexible way to guide independent learning 

• Sees criteria as internalized and uses for self-assessment 



• Creates own criteria 
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This handout accompanies a slide-tape of student examples illustrating this framework of student perspectives. The 
Assessment Committee drew the framework from research on Alverno College students completed by the College's Office 
of Research and Evaluation. 



@ Copyright 1984. Alverno College Productions, Milwaukee, Wisconsin. All rights reserved under U.S., International 
and Universal Copyright Conventions. Reproduction in part or whole by any method is prohibited by law. 
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Appendix D: Alvarno Student*' Developing EBEgPflgtigflg 
on ^Self -Aaseamaent . » «DBing Feedback r « and gfiBMtLtaant tf» 
Improvement* that Lead to Taking Raaponeihilifcy for Learning and 

Bftina Different Waya of Learning. 



Self-Assessment 



• Makes judgments on her own behavior when 

someone else points out concrete 
evidence to her 

• Recognizes that her attitudes affect 

her work 

• Recognizes contradictory evaluations 

of her work 

• Expects the teacher to take the 

initiative in recognizing 
her problems and approaching 
her about them 

• Responds to divergent values with 

self-assessment insights 
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Senses when her own performance in a 
given situation is essentially 
competent or incompetent 
Aware that the learning process 
requires a change in approach 
to learning 
Knows her strengths 
Reflects on a given performance as 
representative of a pattern in her 
own behavior 
Sees criteria as a framework for 

feedback and self-assessment 
Sees criteria as providing a picture 

of the ability to perform 
Compares self to self, rather than 

just self to others 
Achieves sufficient awareness of self to 
assess her own abilities and how they 
contribute to a situation (rather than 
an undifferentiated sense of how "she" 
contributed) 



Using Feedback 
BEGINNING STUDENT 



• At this point f experiences 

evaluation of her 
performance as general 
affirmation or rejection 
of herself 

• Her emotional response to 

evaluation, as of yet, 
Interferes with insight 
into her performance 

• Can connect feedback received 

to subsequent classroom 
experience 



DEVELOPING STUDENT 



• Sees the value in separating 

emotional response to 
feedback from more 
objective stance 

• Sees that feedback on strengths 

and weaknesses provides 
explicit information on 
progress and success 

• Accepts criticism and 

suggestions and follows 
through 



Commitment to Improvement 



• Knows she should improve, wants 

to improve; tries to improve 
in quality ways 

• Recognizes negative attitudes; 

expresses willingness to 
changa 



• Thinks about how to improve 

• Builds on her strengths 

• Sees that criteria given 

ahead of time tell you 
what to learn and what 
to do 

• Motivated to achieve by 

explicit criteria 

• Performs well in structured 

situations; follows 
through if there are 
external demands 

• Completes assignments in 

weak areas; is becoming 
aware of her weaknesses 



13ti 



Appendix D (continued): Alverno students' Developing Perspectives 
on "Self -Assessment." "Usi ng Feedback." and "Comifent to 
T»prQve»enf that Lead to Taking Responsibility for L earning and 

Using Differen t Ways of Learning. 



in 



Self-Assessment 



Sees own abilities apatt from a given 

situation 
Sees abilities as frameworks for 

performing and criteria as a 

picture of the ability for 

performing and self-assessment 
Emphasizes reliance on self-evaluation 

and self-assessment 
Consistently applies self -awareness of 

self (therefore, has more 

knowledge of her abilities— 

acts accordingly) 
Shapes her aspirations realistically, 

commensurate with her abilities 
Gives evidence of Internalizing 

standards of self-assessment 
Sets personal standards out of her 

expectations of her professional 

needs 

Shows Interest in her ability relative 
to other professionals 



Using Feedback 
ADVANCED STUDENT 



Seeks out formative evaluation 
of her work (doesn't just 
wait for someone else's 
summatlve evaluation) 

Self-applies formative 
evaluations of 
her work 

Acts on feedback 

Expects feedback that helps 
her "take charge" 

Expects feedback that helps her 
see patterns and relationships 
to her performance In other 
ability areas 



Commitment to Improvement 



! 



Knows what she needs to do 

to Improve 
Consistently makes an effort 

to Improve processes 
Uses resources to help 

her Improve processes 
Takes Initiative to Improve her 

work 9 finds help when 

she needs It 



The Alverno College Assessment Committee drew this framework from research on Alverno College students completed 
by the College's Office of Research and Evaluation and the Department of Business and Management • 
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Appendix E: Summary of MANOUA Analyses and Associated S tatistics . 
Table 1 

CRITICAL THINKING APPRAISAL 
Summary of Between Effects For MANOUA Analysis 

Age (3) By 

Bv Time UWf Entrance. 2 Years Later. 1 and 1/2 Years Later, 
And * Vft«r-q After Alverno 



Between 

Effects Test Age 
By Subscale (2,132) 



Inferences F 1 

Assumptions F - 1.28 

Deductions F 1 



Table 2 

CRITICAL THINKTMB APPRATSAT. 
Summary of Univariate Effects For MAMMA Analysis* 
Ane <3) By 

Time f 4) /(Entrance. 2 Years Later. 1 and 1/2 Yea rs Later. 
And 5 Years After Alverno 



Effects Through Time 



Time Age By Time 

Univariate Effects Effects 

Tests (1,132) (2,132) 



Linear 

Inferences F ■ 21.80*** F = 2.62 

Assumptions F = 8.78** F = 2.05 

Deductions F = 21.78*** F = 3.86a 

Quadratic 

Inferences F = 1.94 F 1 

Assumptions F 1 F = 2.09 

Deductions F = 1.84 F 1 

Cubic 

Inferences F 1 F 1 

Assumptions F ■ 2.35 F 1 

Deductions F ■ 1.03 F ■ 1.12 



♦ Multivariate F_ tests are significant for all 

significant univariate effects reported, 
a This effect is not significant at the multivariate level. 
** p .01. *** p .001 
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Appendix E (continued) : Summary of MANOVA Analyses and Associated Statistics. 
Table 3 

CRITICAL THINKING APPRAISAL 

Maana and «t-w*«nn* [jsalailsxm aaaopiatafl with 

thl Linear. Effects of Time (4 levels) 



Time of Administration 



Parameter 


At 
Entrance 


One And 
T«o Years A Half 
Later Years later 


Five Years 
After Alverno 






Inferences 






Mean 


9.60 


10.19 


10.37 


10.93 


Std. Dev. 
N 


(3.00) 
(135) 


(3.03) 
(135) 


(3.17) 
(135) 


(3.00) 
(135) 






Assumptions 






Mean 


11.20 


11.09 


11.60 


11.84 


Std. Dev. 
N 


(2.31) 
(135) 


(2.46) 
(135) 


(2.78) 
(135) 


(2.54) 
(135) 






Deductions 






Mean 


16.17 


16.52 


17.14 


17.60 


Std. Dev. 
N 


(3.00) 
(135) 


(3.40) 
(135) 


(3.14) 
(135) 


(3.15) 
(135) 



140 
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Appendix E (continued) : fimmtarv of mamewa AnalvaeB and Associated Statistics . 
Table 4 

DEFINING ISSUES TEST; P% 
Summary of B Atwean Effenta For MANOUA Analysis 
Age m Bv 

Tim« m /(Entrance. 2 Years Later. 1 and 1/2 Years Later, 

3 Yfturft After Alverno 



Age 
(2,88) 



P = 1.76 



Ml 
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Appendix E (continued): Summary of MANOVA Analyses and Associated Statistics . 
Table 5 

DEFINING ISSUES TEST: P% 
Summary of Univariate Effects For MANOVA Analysis* 

Age (3) By 

Time UWf Entrant 7 Ywrc Tafer f l and 1/2 Years Later, 
And 5 Years After Alverno 



Effects Through Time 



Univariate 
Tests 



Time 
Effects 
(1,88) 



Age By Time 
Effects 
(2,88) 



Linear 

Quadratic 

Cubic 



F = 29.18*** F = 1.73 
F = 25.15*** F 1 
F 1 F 1 



+ Multivariate or Average F Statistics are significant for all 

reported Univariate effects. 
* p .05. ** p .01. *** p .001 
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Appendix E (continued): Summary of MANOVA An alyses and Associated Statistics, 
Table 6 

DEPIMTNB ISSUES TEST; P* 
Means and St andard Deviations Associated With the 

Tiirrar ftnfl ttadtatic Time m Effects 



Time of Administration 



One And 

At Two Years A Half Five Years 

Parameter Entrance Later Y**™ T^At After Alverno 



Mean 38.79 46.30 49.92 50.38 

Std. Dev. (13.78) (15.03) (14.20) (14.52) 

N (91) (91) (91) (91) 
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Appendix E (continued) : Summary of MANOVA Ana lyses and Associated Statistics . 
Table 7 

SENTENCE COMPLETION TEST OF F»Q nEWRTOPfllgMT 1 
Summary of Between Effects F or MANOVA Analysis 

Time I 4) / (Entrant 7 Vgarp. Tifltftr, 1 and 1/2 Years Later. 
And 5 Years After Alverno 



Age 
(2,148) 



F = 1.13 



144 
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Appendix E (continued) : Summary of 



Analyses and Associated Statistics. 



Table 8 

SEWTEMCE COMPLETION TEST OF EGO nFVETOPMENT 
Summary of Univariate Effects For MAMOVA Analysis* 

MWfEntrancg. 2 Years Later. 1 and 1/2 Years Later. 
And 5 YAara After Alverna 



Effects Through Time 



Univariate 
Tests 



Time 
Effects 
(1.1481 



Age By 
Time Effects 

(2.1481 



Linear 



F = 18.52*** 



F 



Quadratic 



F = 16.10*** 



F 



1 



Cubic 



F = 1.67 



F 



1 



+ Multivariate £ Statistics are significant for all 

reported univariate effects. 
*** p .001 
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Appendix E (continued) : Summary of MANCMA Analyses >■ id Associated Statistics . 
Table 9 

SENTENCE COMPLETION TEST OF ran nElTET/fflyEWP 
MSflDfi MrvS naviations For th« Mn«mr and Quadratic 

Effect of Time f4 levels) 



Time of Administration 



Parameter 



At 
Entrance 



Two Years 
Later 



One And 
A Half 



Five Years 
After Alverno 



N 



Mean 

Std. Dev. 
U 



5.40 

(.85) 

(151) 



5.31 
(1.06) 
(151) 



5.19 
(1.01) 
(151) 



5.72 
(1.01) 
(151) 



14 B 
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Appendix E (continued) : Summary of MANOVA Analyses and Associated Statistics. 
Table 10 

SENTENCE GO MPLETIQM TEST OF EGO DEVELOPMENT 
Summary o f Between Effects For MAMCVA Analysis 
Age (3) By Mother's Education (2) By 
Time (3) /(Entrance , o Yaars Later And Five Years After Alverno 



Age By 

Mother's Mother's 
Age Education Education 

(2,210) (1,210) (2,210) 



F = 1.84 F 1 F = 1.23 



1 1 7 
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Appendix E (continued) : Summary of MANOVA Analyses and Associated statistics. 
Table 11 

SENTENCE COMPLETION TEST OF EGO DEVELOPMENT 
Summary of Univariate Effects For MAMMA Analysis* 
Agfa (3) By Mother's Education (2) By 
^iF* f3VfEntrft n ™l. 2 v ftfl1ffi Later And F<™> Vftflr-a After Alverno 



Effects Through Time 



Age By 

Mother's Mother's 

Time Age By Education By Education By 

Univariate Effects Time Effects Time Effects Time Effects 

lEfifitfi (1,210) (2,210) (1.210) (2.210) 

Linear F = 18.48*** F 1 F 1 F » 1.24 

Quadratic F = 28.60*** F 1 F 1 F 1 



♦ Multivariate £ Statistics are significant for all 

reported univariate effects. 
*** p .001 
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Appendix F: Summary of BEI Generic Abilities . 
SOCIO-EMOTIONAL MATURITY 

1. Cclf-Control/Emotional Stamina: Appropriately and effectively controls emotions so that they do not disrupt performance, 

2. Ego Strength : Is assertive or perseveres in judgment while withstanding confrontation, disagreement or disapproval. Requires 
element of risk. 

3. Stamina : Perseveres in a potentially or actually stressful situation. 

4. Spontancitv/Curiositv : Playful thought, action or expression. 

5. Perceptual Objectivity : Demonstrates understanding of more than one perspective. 

6. Accurate Self-Assessment : Accurately assesses strengths and weaknesses of own performance to improve or assess own 
abilities. 

7. Reflective Thinking/Valuing: Identifies and reflects upon her own behavior, feelings or beliefs and their consequences. It may 
include reflecting upon a weakness or mistake, and must result in showing or searching for new insight about self or values. 

ENTREPRENEURIAL ABILITIES 

8. Proactivitv: Taking action that is not scripted by the situation or her role and taking responsibility for the outcome/judgment 
(not "We"). 

A. Initiative 

B. Risk taking initiative 

9. Efficiency Orientation : Desire to do something better than a standard of excellence. 

10. Efficiency Actions : Actions toward more efficient use of time and resources. 
INTELLECTUAL ABILITIES 

1 1. Specialized Knowledge : Use of knowledge that takes more than one year to acquire on the job. 



12. Diagnostic Information Seekin g: Seeks information in an ambiguous situation. 

13. Reflectively Coordinated Practice : Receptive to information value of ongoing events: coordinates ongoing actions, relates 
unusual observations or uses new information to adjust ongoing actions or plans. 

14. Pattern recognition: Comparing a stimulus to an understanding or representation stored in memory. 

15. Sees Alternatives : Sees a range of implications, consequences or alternatives, or sees if-then relationships. 

16. Hypothetical/Causal Thinking : Shows systematic understanding of possibility and causality. 

17. Planning/Systematic Thinking : Takes well-ordered and logical approaches to analyzing problems, organizing work and planning 
action. 

18. Conceptualization : Identifying and seeing the key relationships between key issues while understanding the big picture. 
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Appendix F (continued): Summary of BEI Generic Abilities . 
INTERPERSONAL 

19. Formal Communication (demonstrated at time of performance): Effectively communicating in formal writing, speaking or 
planned interactive situation. 



20. Concern With Affiliation : Wanting to be with someone else in order to enjoy mutual friendship or company (warmth 
necessary): Exhibits concern over establishing, maintaining or restoring a warm relationship with another person. This 
relationship is most adequately described by the word friendship. 

21. Affiliative Action: Enjoying, maintaining or establishing companionship for its own sake (warmth necessary). 



22. Positive Regard: Showing respect for others: seeing them as capable and worthy. 

23. Sensitive to Individual Differences: Being aware of and responsive to individual differences. 

24. Accurate Empathy: Effectively reads the moods and feelings of others. 



25. Development of Others: Use of a variety of strategies to insure others' development and to improve their performance. 

26. Development of Self : Develops own knowledge, skills or capability. 

27. Concern With Impact : Exhibits concern about establishing, maintaining or restoring impact, control or influence over other(s) 
(beyond routine and not avoiding power). 

28. Use of Informational/Expert Influence: Uses own credentials for knowledge, access to specific information or construction of 
rationale to persuade others, including building support for ideas/objectives. 

29. Developing Or Using Relational Power: Establishes a warm or inclusive relationship (or uses others' desire for one) in order 
to influence behavior of others. 



30. Use of Socialized/Political Power Effectively working with others to accomplish tasks. 

A. Political Organizational Action 

B. Dyad/Small Group Action 

31* Development of Orga nizational Power : Develops organizational power through strategic personal contact or shows 
understanding of how organization is functionally structured. 

A. Develops Organizational Power 

B. Understands Functional Structure 



32. Use of Unil ateral Power : Gives directions/orders based upon personal authority or rules/procedures to obtain compliant 
behavior. 



© Copyright 1991. Alverno College Productions, Milwaukee, Wisoonsta. All rights reserved under U. S., International 
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w Bft q ard and ftftv ^ o p»ent of Organ i zfltional Power . 



SITUATION: 

I developed and continue to work with 
a computerized system that measures 
the times various nursing activities 
require, and then totals the times as 
nurses entered the activities. 

When the system actually began to 
operate within the hospital and 
staff, who are users of the system 
from the standpoint of being the ones 
that have to enter the data, and the 
system would not be a good system 
unless staff nurses consistently 
entered data and believe that they 
should be accurate with It. And as 
I recognized that staff nurses 
respected the system and had 
confidence In It and said, "This 
system tells what we are doing," as 
opposed to the system we had used In 
the past that had little support of 
the users, that made me feel that It 
wasn't Just that I thought that we 
had done a good Job. That It really 
Is a good system and made me feel It 
Is a system should be publlrhed or It 
should be broadcast . 

As I have been moving along through 
this project of developing this 
computerized data collection system, 
I did keep track of what I did so 
that I would have a lot of minutes of 
meetings. I carefully planned 
meetings so that I always had 
agendas. I worked with a lot of 
different groups of people, and so I 
had a lot of Information that I could 
then look at when I thought about 
writing an article and submitting It 
to the international society. 

I had encouragement from a nursing 
administrator and from the head of a 
technical department who said, "This 
should be written up. This Is an 
excellent system; let's write it up; 
let's get It out there." Probably 
one of the big factors that led to 
actually doing it would have been the 
head of the technical department who 



4 

5" 

6 

7 

8 

9 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 



33- 
34 
35 
36 
37 
33 
39 
40 
41 
42 
43 
44 
41 

47 
48 
49 
50 
51 
52 
53 
54 
55 



Efficiency Actions 
(10B2 and 10B3) 



Development of 
Organizational 
Power 
(31A1) 



ERIC 



151 



Appendix 6 (continued): Example of the Cod ing of Efficiency Actions. 167 
Positive Regard and Development of Organizational Power. 



said that he would be very happy to 
work with me on It. He has a word 
processor and did a lot of the 
mechanics of putting the article 
together. He would take a lot of what 
I had and get that into print so that 
we could start looking at it. That 
was certainly a big reason tor moving 
ahead with it too. 

THOUGHT/FELT: 

When I began I wanted to tell about 
the system. By the time I had 
completed the article, I wanted other 
people to know what the system was, 
and that was more important to me. 
But when I began, I really wanted to 
tell all the things that I had done. 

ACTUALLY DID: 

To Get Started With the Publishing, 
I began by looking at all of the 
information that I had that led up to 
the system. And then I did an 
outline of what I thought others 
would want to hear from within the 
context of the information that I had 
about the system. The article had to 
be shortened considerably because I 
began with taking all of the data 
that related to the various points in 
the outline and then pared it down 
from there. I knew it had to be 
shortened because we had page number 
constraints . 

To Shorten the Article, I started by 
taking Just what, I thought, had to 
be included. The really pertinent 
information: that if that was missing 
it wouldn't make much sense. And I 
tried to keep in mind that if you can 
tell what It is you are doing, why 
you're doing it, what happened when 
you did it, and what are the results 
that you see. By this time it was 
already in pros. I probably had to 
do three revisions before I had it to 
the right length. Again, I had help 
from the head of the technical 
department with the revision process, 
but primarily again it was looking at 
what would be the points that would 
interest people. I did have to let 
go of some of what interested me the 
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Appendix G (continued): Example <* Coding of BfflfllfflWY Actions 
Positive Regard and Beve l OWflnt TTf ft nP*M l int Iff" 1 * 1 Po "** T '- 



most and some of my pet part* that I 
felt were Just absolutely vital to be 
said. But I did make myself look at 
It as if I were a hospital 
administrator. There are people from 
all areas of health care, a lot of 
physicians, and I had to consider 
what would interest them as opposed 
to what was important for the nursing 
staff . 

The head of the technical department 
was very helpful in that he put 
everything on his word processor and 
so as I would revise things, he would 
bring them back to me for further 
review and revision. And he would 
again pull back the latest copy of 
what I thought we should have. Then 
after I agreed that it was the way I 
wanted to submit it, he did do the 
final preparation of putting it onto 
the big papers that you have to use. 
He Just did editing. I did put him 
on as co-author because we had worked 
on the project together. He has 
since published a paper from his 
department's standpoint on the same 
system which he is Just using the 
material that I had developed, and he 
put me in as a co-author. We had 
agreed to that. 

OUTCOME : 

The article that I had submitted to 
an international society that holds 
a conference every three years was 
accepted for publication in the 
proceedings, and then I was also 
asked to present the paper. That was 
very satisfying because 1 felt that 
the system that we have in place is 
really very sophisticated, and I 
personally have felt that it's 
probably one of the best ones that 
I've been able to see anywhere. It 
made me feel that other people at 
least felt that it was a good system 
if they wanted me to come in and 
present it. 

When I presented, 1 had a great deal 
of positive feedback with people from 
other countries even commenting to me 
personally that they felt it was the 
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Appendix G (continued): Kv»mp]ft of the Coding of Efficiency Actions f 
Positive Regard and Development of Organizational Power. 



best presentation of a five day 166 

conference. As I thought about that, 167 

I really believe that the reason I 168 

got that kind of feedback wasn't 169 

because of my presentation, but 170 

because I did stick to being able to 171 

present something that "This Is how 172 

we did It; this is what It Is; this 173 

Is how we did It; these are the 174 

results," whereas many of the 175 

presentations were done simply from 176 

description of a system as opposed to 177 

being able to say "this Is what It 178 

Is; etc." 179 

Note: This write-up was coded for 
many abilities In addition to 
efficiency actions, positive regard, 
and development of organizational 
power. For the sake of clarity, 
those codes are not Included on this 
sample . 
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